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United  States  General  Accounting  Office 
Washington,  DC  20548 


September  14,  2001 

The  Honorable  Wayne  Allard 

Ranking  Minority  Member 

Subcommittee  on  Housing  and  Transportation 

Committee  on  Banking,  Housing,  and  Urban  Affairs 

United  States  Senate 

Dear  Senator  Allard: 

The  accompanying  report  is  the  first  in  a  series  of  reports  responding  to 
your  February  2001  request  that  we  examine  a  range  of  management 
issues  at  the  Department  of  Housing  and  Urban  Development.  This  report 
discusses  our  assessment  of  the  department’s  capability  to  acquire 
software.  We  are  making  recommendations  to  the  Secretary  of  Housing 
and  Urban  Development  to  assist  the  department  in  improving  this 
capability. 

We  are  sending  copies  of  this  report  to  the  Secretary  of  Housing  and 
Urban  Development,  the  Director  of  the  Office  of  Management  and 
Budget,  and  other  congressional  committees.  We  will  also  make  copies 
available  to  other  interested  parties  upon  request. 

If  you  have  questions  or  wish  to  discuss  the  issues  in  this  report,  please 
contact  me  at  (202)  512-6240  or  by  email  at  koontzl@gao.gov.  An 
additional  GAO  contact  and  staff  acknowledgements  are  listed  in 
appendix  II. 

Sincerely  yours, 

Linda  D.  Koontz 

Director,  Information  Management  Issues 
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Executive  Summary 

Purpose 

The  Department  of  Housing  and  Urban  Development  (HUD)  routinely 
acquires  new  information  systems  and  enhancements  to  manage  and 
support  its  various  programs  and  operations.  GAO  has  designated  HUD’s 
major  program  areas  as  high  risk,  in  part  because  the  department’s 
information  and  financial  management  systems  were  poorly  integrated, 
ineffective,  and  generally  unreliable.1  HUD  has  been  endeavoring  to 
improve  its  systems  to  better  support  its  missions  and  management 
reforms. 

Because  of  the  importance  of  information  and  financial  management 
systems  and  related  improvement  efforts  to  meeting  HUD’s  mission,  the 
Ranking  Minority  Member,  Subcommittee  on  Housing  and  Transportation, 
Senate  Committee  on  Banking,  Housing,  and  Urban  Affairs,  requested  that 
GAO  review  the  maturity  of  HUD’s  software  acquisition  processes.  More 
mature  processes  increase  the  likelihood  that  the  software  acquired  will 
meet  an  organization’s  needs. 

Background 

HUD  is  the  principal  federal  agency  responsible  for  housing  and 
community  development  programs.  HUD’s  mission  includes  making 
housing  affordable  (through  mortgage  insurance  for  multifamily  housing 
and  through  rental  assistance),  helping  to  revitalize  localities,  and 
encouraging  home  ownership.  In  fiscal  year  2001,  the  department’s  budget 
was  about  $32  billion,  including  $360  million  for  information  technology 
investments. 

High-quality  software  is  essential  for  HUD’s  information  systems  to 
provide  reliable  management,  financial,  and  administrative  information 
and  support  the  department’s  many  programs.  The  quality  of  software  is 
governed  largely  by  the  quality  of  the  processes  involved  in  developing  or 
acquiring  it  and  in  maintaining  it.  Carnegie  Mellon  University’s  Software 
Engineering  Institute  (SEI),  recognized  for  its  expertise  in  software 
processes,  has  developed  models  and  methods  that  define  and  determine 
organizations’  software  process  maturity.  Together,  these  models  and 

lGAO  High-Risk  Program  (GAO/AIMD-94-72R,  January  27, 1994);  High-Risk  Series: 
Department  of  Housing  and  Urban  Development  (GAO/HR-95-1 1,  February  1995);  High- 
Risk  Series:  Department  of  Housing  and  Urban  Development  (GAO/HR-97-12,  February 
1997);  Major  Management  Challenges  and  Program  Risks:  Department  of  Housing  and 
Urban  Development  (GAO/OCG-99-8,  January  1999);  Major  Management  Challenges  and 
Program  Risks:  Department  of  Housing  and  Urban  Development  (GAO-Ol-248,  January 
2001). 
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methods  provide  a  logical  framework  for  baselining  an  organization’s 
current  process  capabilities  (i.e.,  determining  what  practices  are 
effectively  implemented  (strengths),  not  effectively  implemented 
(weaknesses),  or  contain  mixed  or  inconclusive  evidence  (observations)) 
and  providing  a  structured  plan  for  incremental  process  improvement. 

Using  SEI’s  software  acquisition  capability  maturity  modelSM  and  SEI’s 
software  capability  evaluation  method,2  GAO  analysts  trained  at  SEI 
evaluated  HUD’s  software  acquisition  maturity  in  four  of  seven  key 
process  areas3  that  are  necessary  to  attain  a  “repeatable”  level  of  process 
maturity.  The  “repeatable”  level  of  process  maturity  is  the  second  level  on 
SEFs  five-level  scale.  An  organization  at  the  repeatable  level  of  process 
maturity  has  the  necessary  process  discipline  in  place  to  repeat  earlier 
successes  on  similar  projects.  Organizations  that  do  not  satisfy  the 
requirements  for  the  repeatable  level  are  by  default  judged  to  be  at  the 
“initial”  level  of  maturity.  This  means  that  their  processes  are  immature,  ad 
hoc,  and  sometimes  even  chaotic,  with  few  of  the  processes  defined  and 
success  dependent  mainly  on  the  heroic  efforts  of  individuals. 

In  this  evaluation,  GAO  examined  five  ongoing  projects  selected  by  HUD 
as  being  representative  of  the  department’s  current  software  acquisition 
practices.4  These  were  the  Public  and  Indian  Housing  Information  Center 
system,  the  Real  Estate  Management  System,  the  Resident  Assessment 
Subsystem,  HUD’s  Central  Accounting  and  Program  System,  and  the 
Empowerment  Information  System. 


Results  in  Brief 


HUD  did  not  fully  satisfy  the  requirements  for  any  of  the  “repeatable”  key 
process  areas  we  reviewed.  While  HUD  has  several  software  acquisition 


2  Capability  Maturity  ModelSM  is  the  service  mark  of  Carnegie  Mellon  University,  and  CMM 
is  registered  in  the  U.S.  Patent  and  Trademark  Office.  GAO  used  Software  Acquisition 
Capability  Maturity  Model  (SA-CMM)  Version  1.2  (CMU/SEI-99-TR-002,  April  1999),  the 
latest  version  of  the  model. 

3  The  four  key  process  areas  GAO  evaluated  are  requirements  development  and 
management,  project  management,  contract  tracking  and  oversight,  and  software 
evaluation.  GAO  did  not  evaluate  the  remaining  three  key  process  areas — software 
acquisition  planning,  solicitation,  and  transition  to  support — because  HUD’s  project  teams 
had  not  recently  performed  activities  in  these  areas. 

4  GAO  also  asked  that  the  five  projects  be  (1)  major  efforts  with  large  software  acquisition 
components,  (2)  managed  by  integrated  project  teams,  (3)  at  different  stages  of  the  life 
cycle,  and  (4)  among  HUD’s  best  managed  projects. 
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process  strengths,  GAO  found  a  large  number  of  software  process 
weaknesses  in  all  key  process  areas  evaluated:  requirements  development 
and  management,  project  management,  contract  tracking  and  oversight, 
and  software  evaluation.  As  a  result,  its  processes  for  acquiring  software 
are  immature  and  can  be  characterized  as  ad  hoc,  sometimes  chaotic,  and 
not  repeatable  across  projects.  These  weaknesses  can  lead  to  systems  that 
do  not  meet  the  information  needs  of  management  and  staff,  do  not 
provide  support  for  needed  programs  and  operations,  and  cost  more  and 
take  longer  than  expected  to  complete.  HUD  is  aware  that  it  has  software 
acquisition  weaknesses,  has  stated  its  commitment  to  improving  its 
software  and  system  acquisition  processes,  and  will  begin  a  process 
improvement  effort  in  the  near  future. 


Principal  Finding: 

HUD’s  Software 
Acquisition  Processes 
Are  Immature 

Certain  weaknesses  were  systemic,  recurring  in  most  or  all  of  the  key 
process  areas.  For  example,  HUD  has  no  overall  policy  for  the  acquisition 
of  software;  the  majority  of  project  teams  did  not  develop  plans  to  conduct 
various  activities;  none  of  the  teams  measured  the  status  of  activities;  the 
majority  of  the  teams  did  not  report  regularly  to  management  on  progress 
and  problems;  and  most  project  managers  did  not  require  regular  reports 
on  progress  and  problems.  These  weaknesses  can  lead  to  systems  that  do 
not  meet  HUD’s  information  needs,  do  not  effectively  support  HUD’s 
programs  and  services,  and  cost  more  and  take  longer  to  complete. 

To  reach  the  repeatable  level  of  maturity,  HUD  must  overcome  the  key 
practice  weaknesses  identified  in  this  report.  Table  1  summarizes  the 
strengths  and  weaknesses  for  the  five  projects. 


HUD  did  not  meet  the  requirements  for  the  repeatable  level  of  maturity  in 
the  four  key  process  areas  we  reviewed.  Of  the  310  key  practices  GAO 
evaluated,  50  percent  constituted  strengths  (i.e.,  key  practice  was 
effectively  implemented),  39  percent  were  weaknesses  (i.e.,  key  practice 
was  not  effectively  implemented),  10  percent  were  rated  as  observations, 
and  1  percent  were  not  rated. 
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Table  1 :  Total  Number  of  Key  Process  Area  Strengths,  Weaknesses,  and 
Observations  for  the  Five  Projects 


Key  practice  ratings 

Key  process  area 

Strengths 

Weaknesses  Observations 

Not  rated 

Requirements 
development  and 
management 

27 

31 

12 

0 

Project  management 

45 

25 

10 

0 

Contract  tracking  and 
oversight 

45 

33 

4 

3 

Evaluation 

37 

33 

5 

0 

Total 

154 

122 

31 

3 

HUD  acknowledged  that  it  has  software  acquisition  weaknesses  and  stated 
its  commitment  to  improving  its  software  and  system  acquisition 
processes.  The  department  has  prepared  a  statement  of  work  to  obtain 
contractor  support  to  begin  a  software  process  improvement  effort.  One  of 
the  tasks  the  contract  will  include  is  development  of  a  software  process 
improvement  plan. 

To  successfully  complete  such  an  effort,  HUD’s  plan  must  address  several 
important  issues.  To  be  comprehensive,  this  plan  should  include  the 
results  of  this  review,  measurable  goals  and  time  frames,  estimates  of 
resource  requirements,  and  time  frames  to  reach  the  repeatable  level.  In 
addition,  it  should  address  the  systemic  weaknesses  that  GAO  found. 
Finally,  the  plan  should  include  steps  to  address  the  key  process  areas 
GAO  did  not  review,  as  well  as  steps  to  ensure  that  all  ongoing  and  new 
software  acquisition  projects  adopt  processes  that  meet  the  requirements 
for  the  repeatable  level. 


Recommendations  for 
Executive  Action 


To  improve  HUD’s  software  acquisition  capabilities,  GAO  recommends 
that  the  Secretary  of  Housing  and  Urban  Development  direct  the  HUD 
Chief  Information  Officer  to  develop  and  implement  a  comprehensive  plan 
for  software  acquisition  process  improvement  that  is  based  on  the 
software  capability  results  in  this  report  and  specifies  measurable  goals 
and  time  frames,  sets  priorities  for  initiatives,  estimates  resource 
requirements  (for  trained  staff  and  funding),  and  defines  a  process 
improvement  management  structure. 


Also,  to  address  the  systemic  weaknesses  mentioned  above,  the  plan 
should  contain  steps  to 
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Agency  Comments 


develop  a  comprehensive  policy  for  the  acquisition  of  software, 
require  plans  for  specific  acquisition  activities, 
measure  and  track  the  status  of  activities  performed  by  the  software 
acquisition  teams,  and 

report  regularly  to  management  on  progress  and  problems. 

In  addition,  to  ensure  that  all  aspects  of  software  acquisition  are 
addressed,  we  recommend  that  the  Secretary  direct  the  CIO  to 

assess  HUD’s  maturity  in  the  three  key  process  areas  that  could  not  be 
evaluated  by  GAO  and  include  any  needed  improvement  actions  in  the 
comprehensive  plan  for  software  acquisition  process  improvement; 
ensure  that  all  new  software  acquisition  projects  in  HUD  adopt  processes 
that  satisfy  at  least  SA-CMM  level  2  requirements;  and 
ensure  that  process  improvement  activities  are  initiated  for  all  ongoing 
software  acquisition  projects. 


In  providing  written  comments  on  a  draft  of  our  report,  HUD  agreed  with 
our  assessment  of  its  software  acquisition  processes  and  our 
recommendations  for  strengthening  them,  and  stated  that  our  assessment 
and  recommendations  will  be  addressed  as  part  of  its  process 
improvement  effort.  This  effort  should  help  the  department  improve  its 
ability  to  acquire  systems  that  meet  management  and  staff  information 
needs  and  efficiently  support  programs  and  operations. 
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Chapter  1:  Introduction 


The  Department  of  Housing  and  Urban  Development  (HUD),  established 
in  1965,  has  as  its  primary  mission  ensuring  a  decent,  safe,  and  sanitary 
home  and  suitable  living  environment  for  every  American.  HUD’s  mission 
includes  making  housing  affordable  for  about  4  million  low-income 
households  by  insuring  loans  for  multifamily  rental  housing  and  by 
providing  rental  assistance.  Its  mission  also  includes  helping  to  revitalize 
over  4,000  localities  through  community  development  programs.  The 
department  encourages  homeownership  by  providing,  through  its  Federal 
Housing  Administration,  mortgage  insurance  for  about  7  million 
homeowners  who  otherwise  might  not  have  qualified  for  loans.  HUD  is 
one  of  the  nation’s  largest  financial  institutions,  responsible  for  managing 
about  $508  billion  in  insured  mortgages  and  $570  billion  in  guarantees  of 
mortgage-backed  securities.  HUD’s  budget  in  fiscal  year  2001  was  about 
$32  billion,  including  $360  million  for  information  technology  investments. 


HUD  Relies  on 
Information  Systems 


Like  many  federal  agencies,  HUD  relies  on  its  information  and  financial 
management  systems  to  help  carry  out  its  missions.  However,  past  reports 
by  us  and  others  showed  that  HUD  was  plagued  by  poorly  integrated, 
ineffective,  and  generally  unreliable  information  systems  that  did  not 
satisfy  management  needs  or  provide  adequate  controls.1  We  designated 
HUD’s  program  areas  as  high  risk  in  part  because  of  widespread  problems 
with  information  and  financial  management  systems.  In  1997,  HUD  began 
a  management  reform  effort  designed  to  transform  its  culture,  manage  its 
programs  and  staff  more  effectively,  ultimately  restore  public  trust  in  the 
department,  and  modernize  it  for  the  21st  centuiy. 


In  January  2001,  we  reported  that  HUD  had  been  making  credible  progress 
towards  addressing  its  management  reform  goals.2  However,  we  also 
reported  that  information  and  financial  systems  remained  an  area  of 
management  challenge  because  of  unresolved  issues.  In  addition,  in  its 
March  1,  2001,  financial  statement  audit  report,  the  Office  of  the  HUD 
Inspector  General  listed  two  material  internal  control  weaknesses  related 
to  systems.  The  report  stated  that  HUD  needs  to  (1)  complete 
improvements  to  its  financial  management  systems  to  meet  its  program 
and  financial  management  needs  and  (2)  enhance  the  Federal  Housing 


1  GAO-Ol-248,  Januaiy  2001;  GAO/OCG-99-8,  January  1999;  GAO/HR-97-12,  February  1997; 
GAO/HR-95-11,  February  1995;  GAO/AIMD-94-72R,  January  27, 1994. 

2  GAO-Ol-248,  January  2001. 


Page  7 


GAO-Ol-962  HUD  Information  Systems 


Chapter  1:  Introduction 


Administration’s  information  systems  to  more  effectively  support  its 
business  practices. 

HUD  is  aware  that  information  systems  remain  an  area  of  challenge.  To 
help  address  some  of  the  challenges,  HUD  plans  to  (1)  institute  more 
disciplined  processes  in  its  information  technology  capital  investment 
planning  reviews  (quarterly  reviews  that  assess  how  well  projects  are 
functioning  and  attempt  to  alert  management  to  potential  problems), 

(2)  continue  training  its  project  managers  in  new  project  management 
techniques,  (3)  complete  efforts  to  bring  all  software  products  under 
standard  change  control  and  configuration  management3  and  designate  a 
separate  organization  to  address  configuration  management  issues, 

(4)  improve  HUD’s  end-to-end  testing4  capabilities,  and  (5)  obtain  and 
deploy  new  software  for  project-level  status  reporting. 

To  manage  specific  software  acquisitions,  HUD  uses  integrated  project 
teams.  These  teams  include  representatives  from  the  business 
organization(s)  that  will  use  the  acquired  software  to  do  their  work; 
information  technology  staff  to  oversee  the  contractor  and  review 
contractor  work  products;  and  contract  and  procurement  staff  to  assist 
the  team  in  contract  management.  Policy  and  procedures  for  software 
acquisition  are  the  responsibility  of  HUD’s  Chief  Information  Officer. 


3  Configuration  management  is  a  discipline  applying  technical  and  administrative  direction 
and  surveillance  to  identify  and  document  the  functional  and  physical  characteristics  of 
hardware  or  software,  control  changes  to  these  characteristics,  record  and  report  change 
processing,  and  verify  compliance  of  hardware  or  software  with  specified  requirements. 

4  End-to-end  testing  is  done  to  verify  that  a  defined  set  of  interrelated  systems,  which 
collectively  support  an  organizational  core  business  area  or  function,  interoperate  as 
intended  in  an  operational  environment.  This  includes  not  only  those  systems  owned  and 
managed  by  the  organization  but  also  those  external  systems  with  which  they  interface. 
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The  Software 
Acquisition  Capability 
Maturity  Model 
Provides  a  Means  of 
Assessing  an 
Organization’s  Ability 
to  Manage  Software 
Acquisitions 


The  software  acquisition  capability  maturity  model  (SA-CMM),5  developed 
by  Carnegie  Mellon’s  Software  Engineering  Institute  (SEI),  is  used  to 
measure  an  organization’s  capability  to  manage  the  acquisition  of 
software.  SEI’s  expertise  in  and  methods  for  software  process  assessment 
are  recognized  and  accepted  worldwide  throughout  the  industry.  The 
model  defines  five  levels  of  software  acquisition  maturity.  Each  level  of 
maturity  (except  level  1)  indicates  process  capability  and  identifies  key 
process  areas.  For  a  maturity  level  to  be  achieved,  all  key  process  areas 
related  to  that  level  must  be  implemented  effectively. 

The  first  level  of  process  capability  is  level  2  (referred  to  as  the  repeatable 
level),  where  basic  management  processes  are  established  to  track 
performance,  cost,  and  schedule,  and  the  necessary  discipline  is  in  place 
to  repeat  earlier  successes  on  similar  projects.  Organizations  that  do  not 
effectively  implement  all  key  process  areas  for  the  repeatable  level  are,  by 
default,  at  level  1  or  the  initial  level  of  maturity.  Level  1  processes  can  be 
described  as  immature,  ad  hoc,  and  sometimes  chaotic;  success  in 
software  acquisition  for  these  organizations  is  usually  dependent  upon  the 
ability  and  commitment  of  the  staff  involved.  Figure  1  explains  further  the 
five-level  software  acquisition  model. 


5  Software  Acquisition  Capability  Maturity  Model  (SA-CMM),  Version  1.2,  Software 
Engineering  Institute,  CMU/SEI-99-TR-002  (April  1999). 


Page  9 


GAO-Ol-962  HUD  Information  Systems 


Figure  1 :  SA-CMM  Levels  and  Descriptions 


Continuously 

improving 

process 


Continuous  process  improvement  is  empowered  by 
quantitative  feedback  from  the  process  and  from  piloting 
innovative  ideas  and  technologies. 


Predictable 

process 


Detailed  measures  of  quality  of  the  software  acquisition 
processes,  products,  and  services  are  collected.  The 
software  processes,  products,  and  services  are 
quantitatively  understood  and  controlled. 


Standard 

consistent 

process 


The  acquisition  organization’s  software  acquisition  process 
is  documented,  standardized,  and  established  as  the 
standard  software  acquisition  process.  All  projects  use  an 
approved,  tailored  version  of  the  organization’s  standard 
software  acquisition  process  for  acquiring  their  software 
products  and  services. 


Disciplined 

process 


Basic  project  management  processes  are  established 
to  track  performance,  cost,  and  schedule.  The 
necessary  process  discipline  is  in  place  to  repeat 
earlier  successes  on  projects  in  similar  domains. 


The  software  acquisition  process  is  characterized  as 
ad  hoc,  and  occasionally  even  chaotic.  Few  processes 
are  defined  and  success  depends  on  individual  effort. 


Table  2  identifies  each  of  the  seven  key  process  areas  associated  with  the 
repeatable  level  of  maturity. 
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Table  2:  Key  Process  Areas  for  SA-CMM  Level  2 


SA-CMM  level  2  key 
process  areas 

Description 

Software  acquisition 
planning 

Ensuring  that  reasonable  planning  for  the  software 
acquisition  is  conducted  and  that  all  elements  of  the 
project  are  included. 

Solicitation 

Ensuring  that  award  is  made  to  the  contractor  most 
capable  of  satisfying  the  specified  requirements. 

Requirements  development 
and  management 

Establishing  a  common  and  unambiguous  definition  of 
software  acquisition  requirements  that  is  understood  by 
the  acquisition  team,  system  users,  and  contractor(s). 

Project  management 

Managing  the  activities  of  the  project  office  and 
supporting  contractors)  to  ensure  a  timely,  efficient,  and 
effective  software  acquisition. 

Contract  tracking  and 
oversight 

Ensuring  that  the  software  activities  under  contract  are 
being  performed  in  accordance  with  contract 
requirements  and  that  products  and  services  will  satisfy 
contract  requirements. 

Evaluation 

Determining  that  the  acquired  software  products  and 
services  satisfy  contract  requirements  before  acceptance. 

Transition  to  support 

Providing  for  the  transition  of  the  software  products  being 
acquired  to  their  eventual  support  organization. 

As  established  by  the  model,  each  key  process  area  contains  five  common 
features — commitment  to  perform,  ability  to  perform,  activities 
performed,  measurement  and  analysis,  and  verifying  implementation — that 
indicate  whether  the  implementation  and  institutionalization  of  the  key 
process  area  can  be  effective,  repeatable,  and  lasting.  The  common  feature 
definitions  are  as  follows: 

•  Commitment  to  perform:  This  feature  describes  the  actions  that  the 
organization  takes  to  establish  the  process  and  ensure  that  it  can  endure. 
Key  practices  typically  involve  establishing  organizational  policies  and 
sponsorship. 

•  Ability  to  perform:  This  feature  describes  the  preconditions  that  must 
exist  in  the  project  or  organization  to  competently  implement  the  software 
acquisition  process.  Key  practices  typically  include  assigning 
responsibility  and  providing  training. 

•  Activities  performed:  This  feature  describes  the  roles  and  procedures 
necessary  to  implement  a  key  process  area.  Key  practices  typically  involve 
establishing  plans  and  procedures,  performing  the  work,  tracking  it,  and 
taking  appropriate  management  actions. 

•  Measurement  and  analysis:  This  feature  describes  the  activities 
necessary  to  measure  progress  and  analyze  the  measurements.  Key 
practices  typically  involve  defining  the  measurements  to  be  taken  and  the 
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analyses  to  be  conducted  to  determine  the  status  and  effectiveness  of  the 
activities  performed. 

•  Verifying  implementation:  This  feature  describes  the  steps  the 
organization  must  take  to  ensure  that  project  activities  are  performed  in 
accordance  with  established  processes.  Key  practices  typically  involve 
regular  reviews  by  management. 

Each  common  feature  consists  of  a  number  of  key  practices — specific 
activities  such  as  developing  an  organizational  policy  for  software 
acquisition,  developing  various  plans  for  software  acquisition  activities, 
and  tracking  a  contractor’s  progress.  When  an  organization  is  evaluated 
against  the  SA-CMM,  comparisons  of  actual  performance  against  a  key 
practice  can  result  in  one  of  four  possible  outcomes  or  ratings: 

•  Strength:  The  key  practice  involved  was  effectively  implemented. 

•  Weakness:  The  key  practice  was  ineffectively  implemented  or  was  not 
implemented. 

•  Observation:  The  key  practice  was  evaluated,  but  cannot  be  characterized 
as  a  strength  because  the  (1)  project  team  did  not  provide  sufficient 
evidence  to  support  a  strength  rating  or  (2)  key  practice  was  only  partially 
performed. 

•  Not  rated:  The  key  practice  is  not  relevant  to  the  project  and  was  therefore 
not  rated. 

To  achieve  the  repeatable  level,  HUD  would  have  to  demonstrate  that  the 
key  practices  related  to  that  level  were  implemented  effectively  in  its 
software  acquisition  projects. 


Objective,  Scope,  and 
Methodology 


The  Ranking  Minority  Member  of  the  Subcommittee  on  Housing  and 
Transportation,  Senate  Committee  on  Banking,  Housing,  and  Urban 
Affairs,  requested  that  we  review  HUD’s  software  acquisition  capabilities. 
Our  objective  for  this  review  was  to  determine  the  maturity  of  HUD’s 
software  acquisition  processes. 


To  determine  HUD’s  software  acquisition  process  maturity,  we  applied  the 
Software  Engineering  Institute’s  SA-CMM  using  our  SEI-trained  analysts. 
We  focused  on  the  key  process  areas  necessary  to  obtain  a  repeatable 
level  of  maturity,  the  second  level  of  SEI’s  five-level  model.  We  met  with 
project  managers  and  project  team  members  to  determine  whether  and  to 
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what  extent  they  implemented  each  key  practice  and  to  obtain  relevant 
documentation.  In  accordance  with  the  SEI  model,  for  each  key  process 
area  we  reviewed,6  we  evaluated  HUD’s  institutional  policies  and  practices 
and  compared  project-specific  guidance  and  practices  against  the  required 
key  practices. 

More  specifically,  for  each  key  practice  we  reviewed,  we  compared 
project-specific  documentation  and  practices  against  the  criteria  in  the 
software  acquisition  model.  If  the  project  met  the  criteria  for  the  key 
practice  reviewed,  we  rated  it  as  a  strength.  If  the  project  did  not  meet  the 
criteria  for  the  key  practice  reviewed,  we  rated  it  as  a  weakness.  If  the 
evidence  did  not  support  a  rating  of  a  strength  or  a  weakness,  we  rated  it 
as  an  observation.  If  the  key  practice  was  not  relevant  to  a  project,  we 
stated  that  the  key  practice  was  not  rated. 

We  evaluated  five  HUD  projects,  each  of  which  is  described  below.  We 
asked  HUD  to  select  five  system  acquisition  projects  that  were 
representative  of  HUD  software  acquisition  practices  and  met  the 
following  criteria:  all  were  (1)  major  efforts  with  large  software  acquisition 
components,  (2)  managed  by  integrated  project  teams,  (3)  at  different 
stages  of  the  life  cycle,  and  (4)  among  the  best-managed  acquisitions.  HUD 
selected  the  following  projects: 

•  Public  and  Indian  Housing  Information  Center  system  (PIC) — an  Internet- 
based  system  that  collects  information  about  the  public  housing  inventory 
managed  by  HUD  and  automates  the  processes  for  allocating  program 
funds  to  public  housing  authorities.  It  is  a  central  repository  for 
information  about  public  and  Indian  housing  events  and  program  areas  as 
well  as  for  the  housing  inventory. 

•  Real  Estate  Management  System  (REMS) — a  system  that  provides 
automated  support  to  collect  and  maintain  data  on  multifamily  insured 
and  assisted  properties  and  their  ownership/management  entities  and 
provides  access  to  data  collected  by  HUD  on  physical  and  financial 
assessments. 


6We  evaluated  HUD  on  four  of  the  seven  key  process  areas — requirements  development 
and  management,  project  management,  contract  tracking  and  oversight,  and  software 
evaluation.  We  did  not  evaluate  HUD  on  the  three  other  key  process  areas— software 
acquisition  planning,  solicitation,  and  transition  to  support — because  these  were  not 
recently  performed  for  the  selected  projects. 
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•  Resident  Assessment  Subsystem  (RASS) — a  system  that  collects 
information  on  public  housing  from  residents  and  manages  the  resident 
assessment  process. 

•  HUD  Central  Accounting  and  Program  System  (HUDCAPS)— HUD’s  core 
accounting  and  general  ledger  system.  It  serves  as  the  standard  accounting 
system  for  all  program  areas. 

•  Empowerment  Information  System  (EIS) — an  enterprisewide  data 
warehouse  project  that  is  to  provide  HUD  users  and  business  partners 
with  access  to  reports  and  analytical  information  across  a  large  variety  of 
program  data  sources. 

We  performed  our  work  from  March  2001  through  August  2001  in 
accordance  with  generally  accepted  government  auditing  standards.  We 
provided  a  draft  of  our  report  to  HUD  for  review,  and  the  department’s 
comments  are  addressed  in  chapter  6  and  reprinted  as  appendix  I. 
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HUD  did  not  satisfy  the  criteria  for  the  repeatable  level  of  maturity  in  this 
key  process  area  because  of  the  number  and  severity  of  key  practice 
weaknesses  found.  As  a  result  of  these  weaknesses,  HUD  is  at  greater  risk 
of  acquiring  software  products  that  do  not  meet  mission  needs  and  that 
exceed  cost,  schedule,  and  performance  goals. 

The  purpose  of  requirements  development  and  management  is  to  establish 
and  maintain  a  common  and  unambiguous  definition  of  software 
requirements1  among  the  acquisition  team,  system  users,  and  software 
development  contractor.  Within  this  key  process  area,  the  SA-CMM 
specifies  key  practices  for  each  of  the  five  common  features  that  an 
organization  must  implement  effectively  to  achieve  the  repeatable  level  of 
maturity.  Generally,  these  practices  include  having  a  wntten 
organizational  policy  for  establishing  and  managing  requirements 
allocated  to  software,  documenting  plans  for  the  development  and 
management  of  requirements,  having  documented  processes  for 
requirements  development,  including  elicitation,  analysis,  and  verification, 
measuring  and  reporting  on  the  status  of  activities  to  management, 
appraising  the  impact  of  system-level  requirements  changes  on  the 
software,  and  having  a  mechanism  to  ensure  that  contractor-delivered 
work  products  meet  the  specified  requirements. 

Of  the  70  key  practices  for  this  key  process  area,  HUD  had  27  strengths,  31 
weaknesses  and  12  observations.  For  the 

•  Commitment  to  perform  feature,  there  were  two  strengths,  five 
weaknesses,  and  three  observations. 

•  Ability  to  perform  feature,  there  were  nine  strengths,  five  weaknesses,  and 
one  observation. 

•  Activities  performed  feature,  there  were  ten  strengths,  14  weaknesses,  and 
six  observations. 

•  Measurement  and  analysis  feature,  there  were  one  strength,  three 
weaknesses,  and  one  observation. 

•  Verifying  implementation  feature,  there  were  five  strengths,  four 
weaknesses,  and  one  observation. 

In  addition,  none  of  the  projects  had  strengths  in  all  the  key  practices. 


1  Requirements  describe  the  functions  that  the  software  will  perform  to  meet  user  needs 
and  provide  support  for  business  processes. 
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As  a  result  of  these  weaknesses,  HUD  is  exposed  to  increased  risks  that 
acquired  software  will  not  meet  its  needs  and  that  projects  will  exceed 
cost,  schedule,  and  performance  goals.  HUD  acknowledges  the  need  to 
improve  its  requirements  development  and  management  processes  and  is 
committed  to  improving  its  software  and  systems  acquisition  capability. 
The  department  has  prepared  a  statement  of  work  to  obtain  contractor 
support  to  begin  a  software  process  improvement  effort.  This  document 
specifies  that  the  contractor  would  review  HUD  software  acquisition 
processes,  prioritize  the  key  processes  and  practices  to  address,  and 
develop  a  plan  to  improve  HUD’s  acquisition  processes. 

Details  of  our  evaluation  are  provided  in  the  following  figure  and  tables. 
Figure  2  provides  a  comprehensive  listing  of  the  strengths,  weaknesses, 
and  observations  for  the  requirements  development  and  management  key 
process  area.  The  specific  findings  supporting  the  practice  ratings  cited  in 
figure  2  are  in  tables  3  through  7. 
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Figure  2:  Requirements  Development  and  Management  Summary 
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A  group  is  responsible  for  performing  requirements 

development  and  management  activities. _ 

Adequate  resources  are  provided  for  requirements 

development  and  management  activities. _ 

Individuals  performing  requirements  development  and 
management  activities  have  experience  or  receive 
training.  _ 


The  project  team  performs  its  activities  in  accordance 
with  its  documented  requirements  development  and 

management  plans. _ 

The  project  team  develops,  baselines,  and  maintains 
software-related  contractual  requirements  and  places 
them  under  change  control  early  in  the  project,  but  not 

later  than  the  release  of  the  solicitation  package. _ 
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The  project  team  appraises  changes  to  the  software- 
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performance,  architecture,  supportability,  system 
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Bidirectional  traceability  between  the  contractual 
requirements  and  the  contractor  team’s  software 
products  and  services  is  maintained  throughout  the 

effort. _ 

The  end  user  and  other  affected  groups  are  involved  in 
the  development  of  all  software- related  contractual 
requirements  and  any  subsequent  change  activity. 
Measurements  are  made  and  used  to  determine  the 
status  of  the  requirements  development  and 

management  activities  and  resultant  products. _ 

Requirements  development  and  management  activities 
are  reviewed  by  the  acquisition  organization  on  a 

periodic  basis. _ 

Requirements  development  and  management  activities 
are  reviewed  by  the  project  manager  on  both  a  periodic 
and  an  event-driven  basis. 
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Key  Strength:  Key  practice  effectively  implemented. 

Weakness  Weakness:  Key  practice  not  effectively  implemented. 

Observation  Observation:  Key  practice  evaluated;  evidence  not  sufficient  to  rate  as  a  strength,  or  practice  only  partially  performed. 
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Table  3:  Requirements  Development  and  Management  Findings  for  the  Public  and  Indian  Housing  Information  Center  System 

Public  and  Indian  Housing  Information  Center  system 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  to  perform  1 

The  acquisition  organization  has  a  written 
policy  for  establishing  and  managing  the 
software-related  contractual 
requirements. 

HUD  has  no  written  policy  for  establishing 
and  managing  software-related 
contractual  requirements. 

Weakness 

Commitment  to  perform  2 

Responsibility  has  been  designated  for 
requirements  development  and 
management  activities. 

Responsibility  for  requirements 
development  and  management  has  not 
been  designated. 

Weakness 

Ability  to  perform  1 

A  group  is  responsible  for  performing 
requirements  development  and 
management  activities. 

No  group  is  assigned  responsibility  for 
performing  requirements  development 
and  management. 

Weakness 

Ability  to  perform  2 

Adequate  resources  are  provided  for 
requirements  development  and 
management  activities. 

There  is  no  mechanism  to  identify 
resource  needs  and  ensure  that  they  are 
provided  to  the  project.  Team  members 
indicated  that  they  do  not  have  adequate 
resources  for  performing  these  activities. 

Weakness 

Ability  to  perform  3 

Individuals  performing  requirements 
development  and  management  activities 
have  experience  or  receive  training. 

Project  team  members  performing 
requirements  development  and 
management  activities  have  experience. 

Strength 

Activity  performed  1 

The  project  team  performs  its  activities  in 
accordance  with  its  documented 
requirements  development  and 
management  plans. 

There  is  no  requirements  development 
and  management  plan,  and  so  the 
activities  are  not  performed  in  accordance 
with  it. 

Weakness 

Activity  performed  2 

The  project  team  develops,  baselines, 
and  maintains  software-related 
contractual  requirements  and  places  them 
under  change  control  early  in  the  project, 
but  not  later  than  the  release  of  the 
solicitation  package. 

The  project  team  developed  and 
baselined  software-related  requirements 
before  the  solicitation  package  was 
released.  The  project  team  provided  no 
evidence  of  change  control. 

Observation 

Activity  performed  3 

The  project  team  appraises  system 
requirements  change  requests  for  their 
impact  on  the  software  being  acquired. 

The  project  team  does  not  assess  system 
requirements  change  requests  for  their 
impact  on  the  software  being  acquired. 

Weakness 

Activity  performed  4 

The  project  team  appraises  changes  to 
the  software-related  contractual 
requirements  for  their  impact  on 
performance,  architecture,  supportability, 
system  resource  utilization,  and  contract 
schedule  and  cost. 

The  project  team  does  not  assess  the 
impact  of  software-related  contractual 
requirements  changes  on  performance, 
architecture,  supportability,  and  resource 
utilization.  Cost  and  schedule  impacts  are 
assessed. 

Weakness 

Activity  performed  5 

Bidirectional  traceability  between  the 
contractual  requirements  and  the 
contractor  team’s  software  products  and 
services  is  maintained  throughout  the 
effort. 

The  project  team  maintains  traceability 
between  the  contract  deliverables  and  the 
contractor’s  task/subtask  specification 
numbering.  Traceability  to  HUD’s  original 
requirements,  however,  is  not  maintained. 

Observation 

Activity  performed  6 

The  end  user  and  other  affected  groups 
are  involved  in  the  development  of  all 
software-related  contractual  requirements 
and  any  subsequent  change  activity. 

End  users  are  involved  in  identifying 
software-related  requirements  for 
development  but  are  not  involved  in 
subsequent  change  activities. 

Observation 
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Public  and  Indian  Housing  Information  Center  system 

Common  feature 

Key  practice 

Finding  Rating 

Measurement  and  analysis  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  requirements 
development  and  management  activities 
and  resultant  products. 

No  measurements  are  made  of  HUD’s  Weakness 

requirements  development  and 
management  activities  and  resultant 
products. 

Verifying  implementation  1 

Requirements  development  and 
management  activities  are  reviewed  by 
the  acquisition  organization  on  a  periodic 
basis. 

Project  status,  including  requirements  Strength 

development  and  management  activities, 
is  reviewed  quarterly  by  acquisition 
management. 

Verifying  implementation  2 

Requirements  development  and 
management  activities  are  reviewed  by 
the  project  manager  on  both  a  periodic 
and  an  event-driven  basis. 

The  project  manager  is  not  briefed  on  Weakness 

project  activities  and  the  project 
manager’s  reviews  are  not  documented. 
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Table  4:  Requirements  Development  and  Management  Findings  for  the  Real  Estate  Management  System 

Real  Estate  Management  System 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  to  perform  1 

The  acquisition  organization  has  a 
written  policy  for  establishing  and 
managing  the  software-related 
contractual  requirements. 

HUD  has  no  written  policy  for  establishing 
and  managing  software-related  contractual 
requirements. 

Weakness 

Commitment  to  perform  2 

Responsibility  has  been  designated  for 
requirements  development  and 
management  activities. 

Responsibility  for  requirements 
development  and  management  activities  is 
designated  in  the  project  plan. 

Strength 

Ability  to  perform  1 

A  group  is  responsible  for  performing 
requirements  development  and 
management  activities. 

End  users  and  project  staff  are  responsible 
for  performing  requirements  development 
and  management  activities. 

Strength 

Ability  to  perform  2 

Adequate  resources  are  provided  for 
requirements  development  and 
management  activities. 

There  is  no  mechanism  to  identify  resource 
needs  and  ensure  that  they  are  provided  to 
the  project.  The  project  manager  said  that 
the  team  does  not  have  adequate 
resources  for  performing  these  activities. 

Weakness 

Ability  to  perform  3 

Individuals  performing  requirements 
development  and  management 
activities  have  experience  or  receive 
training. 

Project  team  members  performing 
requirements  development  and 
management  activities  have  experience. 

Strength 

Activity  performed  1 

The  project  team  performs  its  activities 
in  accordance  with  its  documented 
requirements  development  and 
management  plans. 

There  is  no  requirements  development  and 
management  plan,  and  so  the  activities  are 
not  performed  in  accordance  with  it.  The 
team  does  follow  a  process  for 
requirements  gathering. 

Observation 

Activity  performed  2 

The  project  team  develops,  baselines, 
and  maintains  software-related 
contractual  requirements  and  places 
them  under  change  control  early  in  the 
project,  but  not  later  than  the  release 
of  the  solicitation  package. 

The  project  team  did  not  baseline 
requirements  before  solicitation.  Task 
orders  do  not  show  traceability  to  software- 
related  requirements. 

Weakness 

Activity  performed  3 

The  project  team  appraises  system 
requirements  change  requests  for  their 
impact  on  the  software  being  acquired. 

The  project  team  does  not  appraise  system 
requirements  change  requests  for  their 
impact  on  the  software  being  acquired. 

Weakness 

Activity  performed  4 

The  project  team  appraises  changes 
to  the  software-related  contractual 
requirements  for  their  impact  on 
performance,  architecture,  support- 
ability,  system  resource  utilization,  and 
contract  schedule  and  cost. 

The  project  team  does  not  assess  the 
impact  of  software-related  contractual 
requirements  changes  on  performance, 
architecture,  supportability,  and  resource 
utilization.  Cost  and  schedule  are 
assessed. 

Weakness 

Activity  performed  5 

Bidirectional  traceability  between  the 
contractual  requirements  and  the 
contractor  team’s  software  products 
and  services  is  maintained  throughout 
the  effort. 

The  project  team  does  not  maintain 
bidirectional  traceability  between  the 
contractual  requirements  and  contractor 
work  products. 

Weakness 
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Real  Estate  Management  System 

Common  feature 

Key  practice 

Finding 

Rating 

Activity  performed  6 

The  end  user  and  other  affected 
groups  are  involved  in  the 
development  of  all  software-related 
contractual  requirements  and  any 
subsequent  change  activity. 

End  users  are  involved  in  the  development 
of  and  changes  to  software-related 
contractual  requirements. 

Strength 

Measurement  and  analysis  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the 
requirements  development  and 
management  activities  and  resultant 
products. 

No  measurements  are  made  of  HUD’s 
requirements  development  and 
management  activities  and  resultant 
products. 

Weakness 

Verifying  implementation  1 

Requirements  development  and 
management  activities  are  reviewed 
by  the  acquisition  organization  on  a 
periodic  basis. 

Requirements  development  and 
management  activities  are  not  routinely 
reviewed  by  the  acquisition  organization. 
However,  the  project  manager  is  a  senior 
official  of  the  acquisition  organization. 

Observation 

Verifying  implementation  2 

Requirements  development  and 
management  activities  are  reviewed 
by  the  project  manager  on  both  a 
periodic  and  an  event-driven  basis. 

The  project  manager  told  us  that  he 
participates  in  biweekly  status  meetings 
that  include  a  review  of  requirements 
development  and  management  issues.  The 
status  meetings  are  documented. 

Strength 
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Table  5:  Requirements  Development  and  Management  Findings  for  the  Resident  Assessment  Subsystem 

Resident  Assessment  Subsystem 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  to  perform  1 

The  acquisition  organization  has  a 
written  policy  for  establishing  and 
managing  the  software-related 
contractual  requirements. 

HUD  has  no  written  policy  for  establishing  and 
managing  software-related  contractual 
requirements.  The  project  team  used  HUD’s 
software  development  methodology  as  its 
standard  for  establishing  and  managing 
software-related  contractual  requirements. 

Observation 

Commitment  to  perform  2 

Responsibility  has  been  designated 
for  requirements  development  and 
management  activities. 

Responsibility  for  requirements  development 
and  management  is  designated  in  the  project 
plan. 

Strength 

Ability  to  perform  1 

A  group  is  responsible  for  performing 
requirements  development  and 
management  activities. 

The  project  team  is  responsible  for  performing 
requirements  development  and  management 
activities. 

Strength 

Ability  to  perform  2 

Adequate  resources  are  provided  for 
requirements  development  and 
management  activities. 

There  is  no  mechanism  to  identify  resource 
needs  and  ensure  that  they  are  provided  to  the 
project.  However,  the  project  manager  stated 
that  they  do  have  adequate  resources  for 
performing  requirements  development  and 
management  activities. 

Strength 

Ability  to  perform  3 

Individuals  performing  requirements 
development  and  management 
activities  have  experience  or  receive 
training. 

Project  team  members  performing 
requirements  development  and  management 
activities  have  experience  and  have  received 
training. 

Strength 

Activity  performed  1 

The  project  team  performs  its 
activities  in  accordance  with  its 
documented  requirements 
development  and  management 
plans. 

The  project  team  performs  its  activities  in 
accordance  with  its  requirements  development 
and  management  plan. 

Strength 

Activity  performed  2 

The  project  team  develops, 
baselines,  and  maintains  software- 
related  contractual  requirements  and 
places  them  under  change  control 
early  in  the  project,  but  not  later  than 
the  release  of  the  solicitation 
package. 

The  project  team  developed  and  baselined 
contract  requirements  in  its  functional 
requirements  document  and  requirements 
traceability  matrix,  and  placed  the 
requirements  under  change  control. 

Strength 

Activity  performed  3 

The  project  team  appraises  system 
requirements  change  requests  for 
their  impact  on  the  software  being 
acquired. 

The  project  team  assesses  system 
requirements  change  requests  for  their  impact 
on  the  software  being  acquired. 

Strength 

Activity  performed  4 

The  project  team  appraises  changes 
to  the  software-related  contractual 
requirements  for  their  impact  on 
performance,  architecture,  support- 
ability,  system  resource  utilization, 
and  contract  schedule  and  cost. 

The  project  team  appraises  changes  to  the 
software-related  contractual  requirements  for 
impact  on  performance,  architecture,  support- 
ability,  system  resource  utilization,  and 
contract  schedule  and  cost. 

Strength 
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Resident  Assessment  Subsystem 

Common  feature 

Key  practice 

Finding 

Rating 

Activity  performed  5 

Bidirectional  traceability  between  the 
contractual  requirements  and  the 
contractor  team’s  software  products 
and  services  is  maintained 
throughout  the  effort. 

The  project  team  maintains  bidirectional 
traceability  between  contractual  requirements 
and  contractor  work  products. 

Strength 

Activity  performed  6 

The  end  user  and  other  affected 
groups  are  involved  in  the 
development  of  all  software-related 
contractual  requirements  and  any 
subsequent  change  activity. 

End  users  are  involved  in  the  development  of 
contractual  requirements,  generation  of  system 
change  requests,  and  testing. 

Strength 

Measurement  and  analysis  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the 
requirements  development  and 
management  activities  and  resultant 
products. 

Masurements  are  only  made  of  HUD’s 
requirements  development  and  management 
products. 

Observation 

Verifying  implementation  1 

Requirements  development  and 
management  activities  are  reviewed 
by  the  acquisition  organization  on  a 
periodic  basis. 

Requirements  development  and  management 
activities  are  not  routinely  reviewed  by  the 
acquisition  organization.  However,  cost  and 
schedule  performance  is  reviewed  periodically 
by  the  acquisition  organization. 

Weakness 

Verifying  implementation  2 

Requirements  development  and 
management  activities  are  reviewed 
by  the  project  manager  on  both  a 
periodic  and  an  event-driven  basis. 

The  project  manager  is  not  briefed  on 
requirements  development  and  management 
activities,  and  the  project  manager’s  reviews 
are  not  documented. 

Weakness 
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Table  6:  Requirements  Development  and  Management  Findings  for  HUD’s  Central  Accounting  and  Program  System 

HUD’s  Central  Accounting  and  Program  System 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  to  perform  1 

The  acquisition  organization  has  a 
written  policy  for  establishing  and 
managing  the  software-related 
contractual  requirements. 

HUD  has  no  written  policy  for  establishing 
and  managing  the  software-related 
contractual  requirements.  The  project  team 
used  HUD’s  software  development 
methodology  as  its  standard  for  establishing 
and  managing  software-related  contractual 
requirements. 

Observation 

Commitment  to  perform  2 

Responsibility  has  been  designated  for 
requirements  development  and 
management  activities. 

The  project  team  has  been  designated 
responsible  for  requirements  development 
and  management,  with  no  specific 
designation  of  what  group  or  person  within 
the  team. 

Observation 

Ability  to  perform  1 

A  group  is  responsible  for  performing 
requirements  development  and 
management  activities. 

The  project  manager  told  us  that  the  project 
team  is  responsible  for  requirements 
development  and  management  activities. 

Observation 

Ability  to  perform  2 

Adequate  resources  are  provided  for 
requirements  development  and 
management  activities. 

There  is  no  mechanism  to  identify  resource 
needs  and  ensure  that  they  are  provided  to 
the  project.  The  project  team  indicated  that 
they  do  not  have  adequate  resources  for 
performing  requirements  development  and 
management  activities. 

Weakness 

Ability  to  perform  3 

Individuals  performing  requirements 
development  and  management 
activities  have  experience  or  receive 
training. 

Project  team  members  performing 
requirements  development  and 
management  activities  have  experience  and 
have  received  training. 

Strength 

Activity  performed  1 

The  project  team  performs  its  activities 
in  accordance  with  its  documented 
requirements  development  and 
management  plans. 

There  is  no  requirements  development  and 
management  plan,  and  so  the  activities  are 
not  performed  in  accordance  with  it. 

Weakness 

Activity  performed  2 

The  project  team  develops,  baselines, 
and  maintains  software-related 
contractual  requirements  and  places 
them  under  change  control  early  in  the 
project,  but  not  later  than  the  release  of 
the  solicitation  package. 

The  project  team  developed  and  baselined 
software-related  contractual  requirements 
before  the  contractor  began  work,  and  has 
maintained  the  requirements  under 
configuration  board  control. 

Strength 

Activity  performed  3 

The  project  team  appraises  system 
requirements  change  requests  for  their 
impact  on  the  software  being  acquired. 

The  project  team  assesses  all  system 
requirements  change  requests  for  their 
impact  on  the  software  being  acquired. 

Strength 

Activity  performed  4 

The  project  team  appraises  changes  to 
the  software-related  contractual 
requirements  for  their  impact  on 
performance,  architecture,  support- 
ability,  system  resource  utilization,  and 
contract  schedule  and  cost. 

The  project  team  does  not  assess  the 
impact  of  software-related  contractual 
requirements  changes  on  performance, 
architecture,  supportability,  and  system 
resource  utilization.  Cost  is  appraised. 

Weakness 
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HUD’s  Central  Accounting  and  Program  System 

Common  feature 

Key  practice 

Finding 

Rating 

Activity  performed  5 

Bidirectional  traceability  between  the 
contractual  requirements  and  the 
contractor  team's  software  products 
and  services  is  maintained  throughout 
the  effort. 

The  project  team  maintains  bidirectional 
traceability  between  the  contractual 
requirements  and  the  contractor  team’s 
software  products  and  services. 

Strength 

Activity  performed  6 

The  end  user  and  other  affected  groups 
are  involved  in  the  development  of  all 
software-related  contractual 
requirements  and  any  subsequent 
change  activity. 

End  users  and  an  independent  verification 
and  validation  team  were  involved  in  testing 
of  changes.  No  evidence  was  provided  to 
document  user  activity  in  the  development 
of  requirements. 

Observation 

Measurement  and  analysis  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the 
requirements  development  and 
management  activities  and  resultant 
products. 

No  measurements  are  made  of  HUD’s 
requirements  development  and 
management  activities  or  resultant  products. 

Weakness 

Verifying  implementation  1 

Requirements  development  and 
management  activities  are  reviewed  by 
the  acquisition  organization  on  a 
periodic  basis. 

Acquisition  management  is  regularly 
updated  on  project  status,  including 
requirements  development  and 
maintenance  activities. 

Strength 

Verifying  implementation  2 

Requirements  development  and 
management  activities  are  reviewed  by 
the  project  manager  on  both  a  periodic 
and  an  event-driven  basis. 

Project  manager  has  regular,  documented 
meetings  to  review  requirements 
development  and  management  activities. 

Strength 
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Table  7:  Requirements  Development  and  Management  Findings  for  the  Empowerment  Information  System 

Empowerment  Information  System 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  to  perform  1 

The  acquisition  organization  has  a 
written  policy  for  establishing  and 
managing  the  software-related 
contractual  requirements. 

HUD  has  no  written  policy  for 
establishing  and  managing  software- 
related  contractual  requirements. 

Weakness 

Commitment  to  perform  2 

Responsibility  has  been  designated  for 
requirements  development  and 
management  activities. 

Responsibility  for  requirements 
development  and  management  has  not 
been  designated. 

Weakness 

Ability  to  perform  1 

A  group  is  responsible  for  performing 
requirements  development  and 
management  activities. 

The  project  team  is  responsible  for 
performing  these  activities. 

Strength 

Ability  to  perform  2 

Adequate  resources  are  provided  for 
requirements  development  and 
management  activities. 

There  is  no  mechanism  to  identify 
resource  needs  and  ensure  that  they 
are  provided  to  the  project.  The  project 
team  stated  that  they  do  not  have 
adequate  resources  for  performing 
requirements  development  and 
management  activities. 

Weakness 

Ability  to  perform  3 

Individuals  performing  requirements 
development  and  management  activities 
have  experience  or  receive  training. 

Project  team  members  have  training  in 

IT  project  management,  including 
requirements  management. 

Strength 

Activity  performed  1 

The  project  team  performs  its  activities 
in  accordance  with  its  documented 
requirements  development  and 
management  plans. 

There  is  no  requirements  development 
and  management  plan,  and  so  the 
activities  are  not  performed  in 
accordance  with  it. 

Weakness 

Activity  performed  2 

The  project  team  develops,  baselines, 
and  maintains  software-related 
contractual  requirements  and  places 
them  under  change  control  early  in  the 
project,  but  not  later  than  the  release  of 
the  solicitation  package. 

The  project  team  developed  high-level 
requirements.  Detailed  requirements 
were  developed  by  the  contractor.  No 
evidence  was  provided  regarding 
change  control. 

Observation 

Activity  performed  3 

The  project  team  appraises  system 
requirements  change  requests  for  their 
impact  on  the  software  being  acquired. 

The  project  team  does  not  appraise 
system  requirements  changes  for  their 
impact  on  the  software  being  acquired. 

Weakness 

Activity  performed  4 

The  project  team  appraises  changes  to 
the  software-related  contractual 
requirements  for  their  impact  on 
performance,  architecture,  supportability, 
system  resource  utilization,  and  contract 
schedule  and  cost. 

The  project  team  does  not  assess  the 
impact  of  software- related  contractual 
requirements  changes  on  performance, 
architecture,  supportability,  system 
resource  utilization,  and  contract 
schedule  and  cost. 

Weakness 

Activity  performed  5 

Bidirectional  traceability  between  the 
contractual  requirements  and  the 
contractor  team’s  software  products  and 
services  is  maintained  throughout  the 
effort. 

The  project  team  does  not  maintain 
bidirectional  traceability  between  the 
contractual  requirements  and  contractor 
work  products. 

Weakness 
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Empowerment  Information  System 

Common  feature 

Key  practice 

Finding 

Rating 

Activity  performed  6 

The  end  user  and  other  affected  groups 
are  involved  in  the  development  of  all 
software-related  contractual 
requirements  and  any  subsequent 
change  activity. 

No  end  users  were  involved  in 
requirements  development  or 
subsequent  change  activities. 

Weakness 

Measurement  and  analysis  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  requirements 
development  and  management  activities 
and  resultant  products. 

Measurements  are  made  that  identify 
the  status  of  requirements  development 
and  management  activities. 

Strength 

Verifying  implementation  1 

Requirements  development  and 
management  activities  are  reviewed  by 
the  acquisition  organization  on  a  periodic 
basis. 

Requirements  development  and 
management  activities  are  reviewed 
periodically  by  the  acquisition 
organization. 

Strength 

Verifying  implementation  2 

Requirements  development  and 
management  activities  are  reviewed  by 
the  project  manager  on  both  a  periodic 
and  an  event-driven  basis. 

The  project  manager  told  us  that  he  is 
not  briefed  on  activities,  and  the  project 
manager’s  reviews  are  not  documented. 

Weakness 
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HUD  did  not  meet  the  criteria  for  the  repeatable  level  in  the  project 
management  key  process  area  because  of  the  number  and  severity  of  key 
practice  weaknesses  found.  These  weaknesses  increase  the  risk  that  the 
software  acquisition  project,  the  project  team,  and  supporting  contractors 
will  not  be  adequately  managed,  resulting  in  software  that  does  not  meet 
mission  needs  or  acquisitions  that  do  not  meet  cost,  schedule,  and 
performance  goals. 

The  purpose  of  project  management  is  to  direct  and  oversee  the  activities 
of  the  project  team  and  supporting  contractors  to  ensure  a  timely, 
efficient,  and  effective  software  acquisition.  Within  this  key  process  area, 
the  SA-CMM  specifies  key  practices  for  each  of  the  five  common  features 
that  an  organization  must  implement  effectively  to  achieve  the  repeatable 
level  of  maturity.  Generally,  these  practices  include  having  a  project  team 
organized  to  accomplish  the  project’s  objective;  having  a  written  policy  for 
the  management  of  the  project;  documenting  plans  for  the  activities  of  the 
project  team;  measuring  and  controlling  the  cost,  schedule,  and 
performance  objectives  throughout  the  software  acquisition;  and  reporting 
to  management  periodically  on  the  status  of  project  management 
activities. 

Of  the  80  key  practices  for  this  key  process  area,  HUD  had  45  strengths,  25 
weaknesses,  and  10  observations.  For  the 

•  Commitment  to  perform  feature,  there  were  four  strengths  and  six 
weaknesses. 

•  Ability  to  perform  feature,  there  were  14  strengths,  four  weaknesses,  and 
two  observations. 

•  Activities  performed  feature,  there  were  22  strengths,  seven  weaknesses, 
and  six  observations. 

•  Measurement  and  analysis  feature,  there  were  no  strengths,  four 
weaknesses,  and  one  observation. 

•  Verifying  implementation  feature,  there  were  five  strengths,  four 
weaknesses,  and  one  observation. 

In  addition,  none  of  the  projects  had  strengths  in  all  the  key  practices. 

As  a  result  of  these  weaknesses,  HUD  is  exposed  to  increased  risks  that 
software  acquisition  projects  will  exceed  cost,  schedule,  and  performance 
goals,  and  that  acquired  software  will  not  meet  mission  needs.  HUD 
acknowledges  the  need  to  improve  its  project  management  processes  and 
is  committed  to  improving  its  software  and  system  acquisition  capability. 
The  department  has  prepared  a  statement  of  work  to  obtain  contract 
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support  to  begin  a  software  process  improvement  effort.  This  document 
specifies  that  the  contractor  would  review  HUD  software  acquisition 
processes,  prioritize  the  key  processes  and  practices  to  address,  and 
develop  a  plan  to  improve  HUD’s  acquisition  processes. 

Details  of  our  evaluation  are  provided  in  the  following  figure  and  tables. 
Figure  3  provides  a  comprehensive  listing  of  the  strengths,  weaknesses, 
and  observations  for  the  project  management  key  process  area.  The 
specific  findings  supporting  the  practice  ratings  cited  in  figure  3  are  in 
tables  8  through  12. 


Page  29 


GAO-Ol-962  HUD  Information  Systems 


Chapter  3:  HUD’s  Project  Management 
Processes  Are  Not  Repeatable 


Figure  3:  Project  Management  Summary 
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Table  8:  Project  Management  Findings  for  the  Public  and  Indian  Housing  Information  Center  System 

Public  and  Indian  Housing  Information  Center  system 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  to  perform  1 

The  acquisition  organization  has  a  written 
policy  for  management  of  the  software 
acquisition  project. 

HUD  has  no  written  policy  for  managing 
the  acquisition  of  software  products  and 
services. 

Weakness 

Commitment  to  perform  2 

Responsibility  has  been  designated  for 
project  management  activities. 

The  project  team  told  us  that  project 
management  responsibilities  had  been 
assigned,  but  no  documentation  was 
provided  to  support  this  statement. 

Weakness 

Ability  to  perform  1 

A  group  is  responsible  for  managing 
software  acquisition  activities. 

The  project  team  told  us  that  IT  and 
business  project  leads  and  support  staff 
are  responsible  for  managing  software 
acquisition  activities,  but  no  documentation 
was  provided  to  support  this  statement. 

Observation 

Ability  to  perform  2 

Adequate  resources  for  the  project  team 
are  provided  for  the  duration  of  the 
project. 

There  is  no  mechanism  to  identify  resource 
needs  and  ensure  that  they  are  provided  to 
the  project.  Team  members  indicated  that 
the  team  does  not  have  adequate 
resources  for  conducting  project 
management  activities. 

Weakness 

Ability  to  perform  3 

When  project  trade-offs  are  necessary, 
the  project  manager  is  permitted  to  alter 
the  software  acquisition  cost,  schedule,  or 
performance. 

The  project  manager  is  permitted  to  alter 
schedule  or  performance  to  ensure  that 
annual  project  budgets  are  not  exceeded. 

Strength 

Ability  to  perform  4 

Individuals  performing  software 
acquisition  management  activities  have 
experience  or  receive  training. 

Project  team  members  performing  project 
management  activities  have  experience 
and  project  management  training. 

Strength 

Activity  performed  1 

The  project  team  performs  its  activities  in 
accordance  with  its  documented  software 
acquisition  plans. 

The  project  team  told  us  they  did  develop 
project  management  plans,  but  no 
documentation  was  provided  to  support 
this  statement. 

Observation 

Activity  performed  2 

The  roles,  responsibilities,  and  authority 
of  the  project  function  are  documented, 
maintained,  and  communicated  to 
affected  groups. 

The  project  team  has  not  documented  the 
roles,  responsibilities,  and  authority  of  its 
members. 

Weakness 

Activity  performed  3 

The  project  team’s  commitments,  and 
changes  to  commitments,  are 
communicated  to  affected  groups. 

The  commitments  and  changes  to 
commitments  are  communicated  during 
small  informal  group  meetings,  but  they  are 
not  documented  and  no  records  are  kept. 

Observation 

Activity  performed  4 

The  project  team  tracks  the  cost, 
schedule,  resource,  and  technical  risks  of 
the  project. 

Cost  and  schedule  are  tracked,  but 
resource  and  technical  risks  of  the  project 
are  not  tracked. 

Observation 

Activity  performed  5 

The  project  team  tracks  project  issues, 
status,  execution,  funding,  and 
expenditures  against  project  plans  and 
takes  action. 

The  project  team  only  tracks  and  takes 
action  on  changes  to  the  project  schedule. 

Observation 
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Public  and  Indian  Housing  Information  Center  system 

Common  feature 

Key  practice 

Finding 

Rating 

Activity  performed  6 

The  project  team  implements  a  corrective 
action  system  for  the  identification, 
recording,  tracking,  and  correction  of 
problems  discovered  during  the  software 
acquisition. 

The  project  team  does  not  maintain  a 
corrective  action  system  for  the 
identification,  recording,  tracking,  and 
correction  of  problems  discovered  during 
the  software  acquisition. 

Weakness 

Activity  performed  7 

The  project  team  keeps  its  plans  current 
during  the  life  of  the  project  as  replanning 
occurs,  issues  are  resolved,  and  new 
risks  are  discovered. 

The  project  team  does  not  keep  its  plans 
up  to  date  over  the  life  of  the  project. 

Weakness 

Measurement  and  analysis  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  project 
management  activities  and  resultant 
products. 

No  measurements  are  made  of  HUD’s 
project  management  activities  or  resultant 
products. 

Weakness 

Verifying  implementation  1 

Project  management  activities  are 
reviewed  by  the  acquisition  organization 
on  a  periodic  basis. 

Acquisition  management  reviews  project 
activities  quarterly. 

Strength 

Verifying  implementation  2 

Project  management  activities  are 
reviewed  by  the  project  manager  on  both 
a  periodic  and  an  event-driven  basis. 

The  project  manager  stated  that  while  he 
meets  with  members  of  his  staff  daily  to 
discuss  the  status  of  the  project,  meeting 
results  and  action  items  are  not 
documented. 

Weakness 

Page  32 


GAO-Ol-962  HUD  Information  Systems 


Chapter  3:  HUD’s  Project  Management 

Processes  Are  Not  Repeatable 

Table  9:  Project  Management  Findings  for  the  Real  Estate  Management  System 

Real  Estate  Management  System 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  to  perform  1 

The  acquisition  organization  has  a 
written  policy  for  management  of  the 
software  acquisition  project. 

HUD  has  no  written  policy  for  managing  the 
acquisition  of  software  products  and 
services. 

Weakness 

Commitment  to  perform  2 

Responsibility  has  been  designated  for 
project  management  activities. 

Responsibility  for  project  management 
activities  has  been  assigned  to  the  project 
manager  in  the  project  management  plan. 

Strength 

Ability  to  perform  1 

A  group  is  responsible  for  managing 
software  acquisition  activities. 

The  project  manager,  along  with  business 
and  IT  support  staff,  is  responsible  for 
project  management  activities. 

Strength 

Ability  to  perform  2 

Adequate  resources  for  the  project 
team  are  provided  for  the  duration  of 
the  project. 

There  is  no  mechanism  to  identify  resource 
needs  and  ensure  that  they  are  provided  to 
the  project.  The  project  manager  stated  that 
the  team  does  not  have  adequate  resources 
for  conducting  project  management 
activities. 

Weakness 

Ability  to  perform  3 

When  project  trade-offs  are 
necessary,  the  project  manager  is 
permitted  to  alter  the  software 
acquisition  cost,  schedule,  or 
performance. 

The  project  manager  is  permitted  to  alter 
schedule  or  performance  to  ensure  that 
annual  project  budgets  are  not  exceeded. 

Strength 

Ability  to  perform  4 

Individuals  performing  software 
acquisition  management  activities 
have  experience  or  receive  training. 

The  project  team  told  us  that  individuals 
performing  project  management  activities 
have  experience  and  have  had  training. 
However,  the  team  did  not  provide 
documentation  to  support  this  statement. 

Observation 

Activity  performed  1 

The  project  team  performs  its  activities 
in  accordance  with  its  documented 
software  acquisition  plans. 

Software  acquisition  project  management 
plans  are  used  to  guide  the  activities  of  the 
project  team. 

Strength 

Activity  performed  2 

The  roles,  responsibilities,  and 
authority  of  the  project  function  are 
documented,  maintained,  and 
communicated  to  affected  groups. 

Roles  and  responsibilities  are  defined  in  the 
project  plan;  however,  responsible  team 
members  are  not  named,  which  hinders 
effective  communication  of  this  information. 

Observation 

Activity  performed  3 

The  project  team’s  commitments,  and 
changes  to  commitments,  are 
communicated  to  affected  groups. 

The  commitments,  and  changes  to 
commitments,  are  communicated  to  affected 
groups  through  weekly  status  meetings, 
emails,  and  telephone  conversations. 

Strength 

Activity  performed  4 

The  project  team  tracks  the  cost, 
schedule,  resource,  and  technical 
risks  of  the  project. 

The  project  team  does  not  track  risks 
related  to  cost,  schedule,  resource,  and 
technical  aspects  of  the  project. 

Weakness 

Activity  performed  5 

The  project  team  tracks  project  issues, 
status,  execution,  funding,  and 
expenditures  against  project  plans  and 
takes  action. 

The  project  manager  tracks  project  status, 
execution,  funding,  and  expenditures,  and 
changes  the  scope  or  schedule  if 
necessary. 

Strength 
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Real  Estate  Management  System 

Common  feature 

Key  practice 

Finding 

Rating 

Activity  performed  6 

The  project  team  implements  a 
corrective  action  system  for  the 
identification,  recording,  tracking,  and 
correction  of  problems  discovered 
during  the  software  acquisition. 

The  project  team  does  not  maintain  a 
corrective  action  system  for  the 
identification,  recording,  tracking,  and 
correction  of  problems  discovered  during 
the  software  acquisition.  Minutes  of 
biweekly  status  meetings  identify  action 
items  and  who  is  responsible  for  each  item. 

Observation 

Activity  performed  7 

The  project  team  keeps  its  plans 
current  during  the  life  of  the  project  as 
replanning  occurs,  issues  are 
resolved,  and  new  risks  are 
discovered. 

The  project  team  does  not  keep  its  plans  up 
to  date  over  the  life  of  the  project. 

Weakness 

Measurement  and  analysis  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  project 
management  activities  and  resultant 
products. 

No  measurements  are  made  of  HUD’s 
project  management  activities  or  resultant 
products. 

Weakness 

Verifying  implementation  1 

Project  management  activities  are 
reviewed  by  the  acquisition 
organization  on  a  periodic  basis. 

Acquisition  management  does  not  regularly 
review  project  management  activities. 

Weakness 

Verifying  implementation  2 

Project  management  activities  are 
reviewed  by  the  project  manager  on 
both  a  periodic  and  an  event-driven 
basis. 

The  project  manager  is  involved  with  the 
project,  and  status  meetings  are  held 
regularly;  however,  the  project  manager’s 
reviews  are  not  documented. 

Observation 
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Table  10:  Project  Management  Findings  for  the  Resident  Assessment  Subsystem 


Resident  Assessment  Subsystem 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  to  perform  1 

The  acquisition  organization  has  a 
written  policy  for  management  of  the 
software  acquisition  project. 

HUD  has  no  written  policy  for  managing  the 
acquisition  of  software  products  and 
services.  The  project  team  uses  HUD’s 
software  development  methodology  where 
applicable. 

Weakness 

Commitment  to  perform  2 

Responsibility  has  been  designated  for 
project  management  activities. 

Responsibility  for  project  management 
activities  has  been  assigned  to  the  IT  and 
business  project  leads. 

Strength 

Ability  to  perform  1 

A  group  is  responsible  for  managing 
software  acquisition  activities. 

The  IT  and  business  project  leads  and 
support  staff  are  responsible  for  project 
management  activities. 

Strength 

Ability  to  perform  2 

Adequate  resources  for  the  project 
team  are  provided  for  the  duration  of 
the  project. 

There  is  no  mechanism  to  identify  resource 
needs  and  ensure  that  they  are  provided  to 
the  project.  The  project  manager  stated  that 
the  team  has  adequate  resources  for 
conducting  project  management  activities. 

Strength 

Ability  to  perform  3 

When  project  trade-offs  are  necessary, 
the  project  manager  is  permitted  to  alter 
the  software  acquisition  cost,  schedule, 
or  performance. 

The  project  manager  is  permitted  to  alter 
schedule  or  performance  to  ensure  that 
annual  project  budgets  are  not  exceeded. 

Strength 

Ability  to  perform  4 

Individuals  performing  software 
acquisition  management  activities  have 
experience  or  receive  training. 

Project  team  members  performing  project 
management  activities  have  experience  and 
have  had  training. 

Strength 

Activity  performed  1 

The  project  team  performs  its  activities 
in  accordance  with  its  documented 
software  acquisition  plans. 

Software  acquisition  project  management 
plans  are  used  to  guide  the  activities  of  the 
project  team. 

Strength 

Activity  performed  2 

The  roles,  responsibilities,  and  authority 
of  the  project  function  are  documented, 
maintained,  and  communicated  to 
affected  groups. 

Roles,  responsibilities,  and  authority  are 
defined  in  the  project  plan  and  in  the 
Configuration  Control  Board’s  (CCB)  role 
matrix. 

Strength 

Activity  performed  3 

The  project  team’s  commitments,  and 
changes  to  commitments,  are 
communicated  to  affected  groups. 

The  project  team’s  commitments  and 
changes  to  commitments  regarding  this 
project  are  communicated  to  affected 
groups. 

Strength 

Activity  performed  4 

The  project  team  tracks  the  cost, 
schedule,  resource,  and  technical  risks 
of  the  project. 

A  risk  management  plan  is  maintained  that 
tracks  risks  associated  with  costs,  schedule, 
resource,  and  technical  risks. 

Strength 

Activity  performed  5 

The  project  team  tracks  project  issues, 
status,  execution,  funding,  and 
expenditures  against  project  plans  and 
takes  action. 

The  project  team  uses  contractor  weekly 
and  monthly  status  reports  to  track 
contractor  costs  and  schedule  issues.  Also, 
a  quarterly  project  control  review  is  done  to 
review  cost  and  schedule  status. 

Strength 

Activity  performed  6 

The  project  team  implements  a 
corrective  action  system  for  the 
identification,  recording,  tracking,  and 
correction  of  problems  discovered 
during  the  software  acquisition. 

A  corrective  action  system  has  been 
implemented  for  the  identification,  recording, 
tracking,  and  correction  of  problems 
discovered  during  the  software  acquisition. 

Strength 
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Resident  Assessment  Subsystem 

Common  feature 

Key  practice 

Finding 

Rating 

Activity  performed  7 

The  project  team  keeps  its  plans 
current  during  the  life  of  the  project  as 
replanning  occurs,  issues  are  resolved, 
and  new  risks  are  discovered. 

The  project  plans  are  kept  up  to  date  as 
replanning  occurs,  issues  are  resolved,  and 
new  risks  are  discovered.  Monthly  reports 
are  provided  that  show  changes  to 
scheduled  software  releases  resulting  from 
replanning. 

Strength 

Measurement  and  analysis  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  project 
management  activities  and  resultant 
products. 

Measurements  are  only  made  of  HUD’s 
project  management  products. 

Observation 

Verifying  implementation  1 

Project  management  activities  are 
reviewed  by  the  acquisition 
organization  on  a  periodic  basis. 

The  acquisition  organization  reviews  project 
management  activities  on  a  quarterly  basis. 

Strength 

Verifying  implementation  2 

Project  management  activities  are 
reviewed  by  the  project  manager  on 
both  a  periodic  and  an  event-driven 
basis. 

The  project  manager  stated  that  while  he 
meets  with  members  of  his  staff  daily  to 
discuss  the  status  of  the  project,  meeting 
results  and  action  items  are  not 
documented. 

Weakness 
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Table  11 :  Project  Management  Findings  for  HUD’s  Central  Accounting  and  Program  System 

HUD’s  Central  Accounting  and  Program  System 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  to  perform  1 

The  acquisition  organization  has  a 
written  policy  for  management  of  the 
software  acquisition  project. 

HUD  has  no  written  policy  for  managing  the 
acquisition  of  software  products  and  services. 

Weakness 

Commitment  to  perform  2 

Responsibility  has  been  designated 
for  project  management  activities. 

Responsibility  for  project  management  activities 
has  been  assigned  to  the  project  lead. 

Strength 

Ability  to  perform  1 

A  group  is  responsible  for  managing 
software  acquisition  activities. 

The  IT  and  business  project  leads  are 
responsible  for  managing  software  acquisition 
activities. 

Strength 

Ability  to  perform  2 

Adequate  resources  for  the  project 
team  are  provided  for  the  duration  of 
the  project. 

There  is  no  mechanism  to  identify  resource 
needs  and  ensure  they  were  made  available  to 
the  project.  The  project  manager  stated  that  the 
team  does  not  have  adequate  resources  for 
performing  project  management  activities. 

Weakness 

Ability  to  perform  3 

When  project  trade-offs  are 
necessary,  the  project  manager  is 
permitted  to  alter  the  software 
acquisition  cost,  schedule,  or 
performance. 

The  project  manager  is  permitted  to  alter  the 
schedule  or  performance  to  ensure  that  annual 
project  budgets  are  not  exceeded. 

Strength 

Ability  to  perform  4 

Individuals  performing  software 
acquisition  management  activities 
have  experience  or  receive  training. 

Project  team  members  performing  project 
management  activities  have  experience  and 
have  had  training. 

Strength 

Activity  performed  1 

The  project  team  performs  its 
activities  in  accordance  with  its 
documented  software  acquisition 
plans. 

Software  acquisition  project  management  plans 
are  used  to  guide  the  activities  of  the  project 
team. 

Strength 

Activity  performed  2 

The  roles,  responsibilities,  and 
authority  of  the  project  function  are 
documented,  maintained,  and 
communicated  to  affected  groups. 

The  roles,  responsibilities,  and  authority  of  the 
project  team  are  defined  in  the  software 
acquisition  project  plan. 

Strength 

Activity  performed  3 

The  project  team’s  commitments,  and 
changes  to  commitments,  are 
communicated  to  affected  groups. 

The  commitments  and  changes  to 
commitments  are  communicated  to  affected 
groups  through  weekly  status  meetings,  emails, 
and  telephone  conversations. 

Strength 

Activity  performed  4 

The  project  team  tracks  the  cost, 
schedule,  resource,  and  technical 
risks  of  the  project. 

The  project  plans  and  exhibits  identify  risks 
associated  with  the  project  and  define 
mitigation  plans  for  these  risks.  However,  the 
project  team  does  not  track  the  risks. 

Weakness 

Activity  performed  5 

The  project  team  tracks  project 
issues,  status,  execution,  funding, 
and  expenditures  against  project 
plans  and  takes  action. 

The  project  team  tracks  project  issues,  status, 
execution,  and  funding  through  weekly 
functional  and  technical  status  meetings. 
Funding  is  tracked  and  actual  costs  compared 
to  projected  costs. 

Strength 

Activity  performed  6 

The  project  team  implements  a 
corrective  action  system  for  the 
identification,  recording,  tracking,  and 
correction  of  problems  discovered 
during  the  software  acquisition. 

A  corrective  action  system  has  been 
implemented  for  the  identification,  recording, 
tracking,  and  correction  of  problems  discovered 
during  the  software  acquisition. 

Strength 
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HUD’s  Central  Accounting  and  Program  System 

Common  feature 

Key  practice 

Finding 

Rating 

Activity  performed  7 

The  project  team  keeps  its  plans 
current  during  the  life  of  the  project 
as  replanning  occurs,  issues  are 
resolved,  and  new  risks  are 
discovered. 

The  project  team  does  have  a  risk  assessment 
and  mitigation  strategy.  However,  the  risks  are 
not  tracked,  and  plans  are  not  kept  current. 

Weakness 

Measurement  and  analysis  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  project 
management  activities  and  resultant 
products. 

No  measurements  are  made  of  HUD’s  project 
management  activities  or  resultant  products. 

Weakness 

Verifying  implementation  1 

Project  management  activities  are 
reviewed  by  the  acquisition 
organization  on  a  periodic  basis. 

The  acquisition  organization  reviews  project 
management  activities  on  a  regular  basis 
through  meetings  with  the  project  manager. 

Strength 

Verifying  implementation  2 

Project  management  activities  are 
reviewed  by  the  project  manager  on 
both  a  periodic  and  an  event-driven 
basis. 

The  project  manager  reviews  project 
management  activities  regularly  through 
briefings  and  document  review. 

Strength 
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Table  12:  Project  Management  Findings  for  the  Empowerment  Information  System 

Empowerment  Information  System 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  to  perform  1 

The  acquisition  organization  has  a 
written  policy  for  management  of  the 
software  acquisition  project. 

HUD  has  no  written  policy  for  managing  the 
acquisition  of  software  products  and  services. 

Weakness 

Commitment  to  perform  2 

Responsibility  has  been  designated 
for  project  management  activities. 

Responsibility  for  project  management  activities 
has  been  assigned  to  the  project  technical  lead. 

Strength 

Ability  to  perform  1 

A  group  is  responsible  for  managing 
software  acquisition  activities. 

The  project  lead  and  assigned  staff  are 
responsible  for  performing  project  management 
activities. 

Strength 

Ability  to  perform  2 

Adequate  resources  for  the  project 
team  are  provided  for  the  duration  of 
the  project. 

There  is  no  mechanism  to  identify  resource 
needs  and  ensure  that  they  are  made  available 
to  the  project.  The  project  manager  stated  that 
the  team  does  not  have  adequate  resources  for 
performing  project  management  activities. 

Weakness 

Ability  to  perform  3 

When  project  trade-offs  are 
necessary,  the  project  manager  is 
permitted  to  alter  the  software 
acquisition  cost,  schedule,  or 
performance. 

The  project  manager  is  allowed  to  alter 
schedule  or  performance  to  ensure  that  annual 
project  budgets  are  not  exceeded. 

Strength 

Ability  to  perform  4 

Individuals  performing  software 
acquisition  management  activities 
have  experience  or  receive  training. 

Project  team  members  performing  project 
management  activities  have  experience  and 
project  management  training. 

Strength 

Activity  performed  1 

The  project  team  performs  its 
activities  in  accordance  with  its 
documented  software  acquisition 
plans. 

Software  acquisition  project  management  plans 
are  used  to  guide  the  activities  of  the  project 
team. 

Strength 

Activity  performed  2 

The  roles,  responsibilities,  and 
authority  of  the  project  function  are 
documented,  maintained,  and 
communicated  to  affected  groups. 

Roles,  responsibilities,  and  authority  of  the 
project  team  are  defined  in  the  project 
organization  chart. 

Strength 

Activity  performed  3 

The  project  team’s  commitments, 
and  changes  to  commitments,  are 
communicated  to  affected  groups. 

The  project  manager  uses  contractor  status 
reports  to  communicate  commitments  and 
changes  to  commitments  to  the  project  team 
and  other  affected  groups  in  meetings. 

Strength 

Activity  performed  4 

The  project  team  tracks  the  cost, 
schedule,  resource,  and  technical 
risks  of  the  project. 

The  project  team  does  assess  and  track  risks 
for  several  areas,  including  contract,  funding, 
schedule,  management,  and  technical  areas. 

Strength 

Activity  performed  5 

The  project  team  tracks  project 
issues,  status,  execution,  funding, 
and  expenditures  against  project 
plans  and  takes  action. 

The  project  team  did  conduct  a  review  of  the 
software  developed  by  the  contractor.  As  a 
result  of  issues  found  during  that  review, 
modifications  were  made  to  the  contract. 

Strength 

Activity  performed  6 

The  project  team  implements  a 
corrective  action  system  for  the 
identification,  recording,  tracking, 
and  correction  of  problems 
discovered  during  the  software 
acquisition. 

A  remedial  action  plan  was  developed  and  a 
corrective  action  system  implemented  to 
record,  track,  and  correct  problems  found 
during  the  software  acquisition. 

Strength 
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Empowerment  Information  System 

Common  feature 

Key  practice 

Finding 

Rating 

Activity  performed  7 

The  project  team  keeps  its  plans 
current  during  the  life  of  the  project 
as  replanning  occurs,  issues  are 
resolved,  and  new  risks  are 
discovered. 

The  replanning  is  documented  in  the  remedial 
action  plan. 

Strength 

Measurement  and  analysis  1 

Measurements  are  made  and  used 
to  determine  the  status  of  the 
project  management  activities  and 
resultant  products. 

No  measurements  are  made  of  HUD’s  project 
management  activities  and  resultant  products. 

Weakness 

Verifying  implementation  1 

Project  management  activities  are 
reviewed  by  the  acquisition 
organization  on  a  periodic  basis. 

The  acquisition  organization  reviews  project 
management  activities  quarterly. 

Strength 

Verifying  implementation  2 

Project  management  activities  are 
reviewed  by  the  project  manager  on 
both  a  periodic  and  an  event-driven 
basis. 

The  project  manager  is  not  briefed  by  other 
members  on  either  a  regular  or  event-driven 
basis,  and  project  manager  reviews  are  not 
documented. 

Weakness 
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HUD’s  contract  tracking  and  oversight  processes  do  not  satisfy  the  criteria 
for  the  repeatable  level  of  maturity  because  of  the  number  and  severity  of 
key  practice  weaknesses  found.  These  weaknesses  increase  the  risk  that 
software  activities  will  not  be  performed  in  accordance  with  contractual 
requirements,  resulting  in  software  acquisitions  being  late,  costing  more 
than  intended,  and  not  performing  as  intended. 

The  purpose  of  contract  tracking  and  oversight  is  to  ensure  that  the 
software  activities  under  contract  are  being  performed  in  accordance  with 
contractual  requirements.  Within  this  key  process  area,  the  SA-CMM 
specifies  a  number  of  key  practices  across  the  five  common  features  that 
an  organization  must  implement  effectively  to  achieve  the  repeatable  level 
of  maturity.  Generally,  these  practices  include  having  a  written 
organizational  policy  for  contract  tracking  and  oversight,  documenting 
plans  for  contract  tracking  and  oversight  activities,  reviewing  contractor 
software  planning  documents  and  using  those  documents  to  oversee  the 
contractor’s  efforts,  comparing  actual  cost  and  schedule  to  planned 
budgets  and  schedules,  and  having  a  mechanism  to  ensure  that  contractor- 
delivered  work  products  meet  contract  requirements. 

For  the  85  key  practices  in  this  key  process  area,  HUD  had  45  strengths,  33 
weaknesses,  four  observations,  and  three  practices  not  rated.  For  the 

•  Commitment  to  perform  feature,  there  were  12  strengths  and  three 
weaknesses. 

•  Ability  to  perform  feature,  there  were  ten  strengths  and  five  weaknesses. 

•  Activities  performed  feature,  there  were  20  strengths,  15  weaknesses,  two 
observations,  and  three  practices  not  rated. 

•  Measurement  and  analysis  feature,  there  were  no  strengths  and  five 
weaknesses. 

•  Verifying  implementation  feature,  there  were  three  strengths,  five 
weaknesses,  and  two  observations. 

In  addition,  none  of  the  projects  had  strengths  in  all  the  key  practices. 

As  a  result  of  these  weaknesses,  HUD  is  exposed  to  increased  risk  that 
software  activities  will  not  be  performed  in  accordance  with  contractual 
requirements,  resulting  in  software  acquisitions  being  late,  costing  more 
than  planned,  and  not  performing  as  intended.  HUD  acknowledges  the 
need  to  improve  its  contract  tracking  and  oversight  processes  and  is 
committed  to  improving  its  software  and  system  acquisition  capability. 
The  department  has  prepared  a  statement  of  work  to  obtain  contractor 
support  to  begin  a  software  process  improvement  effort.  This  document 
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specifies  that  the  contractor  would  review  HUD  software  acquisition 
processes,  prioritize  the  key  processes  and  practices  to  address,  and 
develop  apian  to  improve  HUD’s  acquisition  processes. 

Details  of  our  evaluation  are  provided  in  the  following  figure  and  tables. 
Figure  4  provides  a  comprehensive  listing  of  the  strengths,  weaknesses, 
and  observations  for  the  contract  tracking  and  oversight  key  process  area. 
The  specific  findings  supporting  the  practice  ratings  cited  in  figure  4  are  in 
tables  13  through  17. 
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Figure  4:  Contract  Tracking  and  Oversight  Summary 
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related  contractual  requirements  are  coordinated  with 
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Strength:  Key  practice  effectively  implemented. 

Weakness  Weakness:  Key  practice  not  effectively  implemented. 

Observation  Observation:  Key  practice  evaluated;  evidence  not  sufficient  to  rate  as  a  strength,  or  practice  only  partially  performed. 
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Table  13:  Contract  Tracking  and  Oversight  Findings  for  the  Public  and  Indian  Housing  Information  Center  System 

Public  and  Indian  Housing  Information  Center  system 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  to  perform  1 

The  acquisition  organization  has  a 
written  policy  for  managing  the  contract 
tracking  and  oversight  of  the  software 
acquisition  project. 

HUD  has  no  written  policy  for  contract 
tracking  and  oversight  of  the  acquisition  of 
software  products  and  services,  and  the 
project  team  did  not  establish  one. 

Weakness 

Commitment  to  perform  2 

Responsibility  has  been  designated  for 
contract  tracking  and  oversight 
activities. 

Responsibility  for  contract  tracking  and 
oversight  activities  is  not  designated. 

Weakness 

Commitment  to  perform  3 

The  project  team  includes  contracting 
specialists  in  the  management  of  the 
contract. 

The  project  team  told  us  they  did  include 
contracting  specialists  in  the  management 
of  the  contract,  but  no  documentation  was 
provided  to  support  this  statement. 

Weakness 

Ability  to  perform  1 

A  group  is  responsible  for  managing 
contract  tracking  and  oversight 
activities. 

The  project  team  told  us  there  was  a  group 
assigned  responsibility  for  contract  tracking 
and  oversight  activities,  but  no 
documentation  was  provided  to  support 
this  statement. 

Weakness 

Ability  to  perform  2 

Adequate  resources  are  provided  for 
contract  tracking  and  oversight 
activities. 

There  is  no  mechanism  to  identify  resource 
needs  and  ensure  that  they  are  provided  to 
the  project.  Team  members  indicated  that 
the  resources  for  performing  contract 
tracking  and  oversight  activities  are  not 
adequate. 

Weakness 

Ability  to  perform  3 

Individuals  performing  contract  tracking 
and  oversight  activities  have 
experience  or  receive  training. 

Project  team  members  performing  contract 
tracking  and  oversight  activities  have 
experience. 

Strength 

Activity  performed  1 

The  project  team  performs  its  activities 
in  accordance  with  its  documented 
contract  tracking  and  oversight  plans. 

There  is  no  contract  tracking  and  oversight 
plan;  therefore,  the  activities  are  not 
performed  in  accordance  with  it. 

Weakness 

Activity  performed  2 

The  project  team  reviews  required 
contractor  software  planning 
documents  which,  when  satisfactory, 
are  used  to  oversee  the  contractor 
team’s  software  effort. 

The  contractor  did  not  prepare  software 
planning  documents  that  the  project  team 
could  use  to  oversee  the  contractor’s 
software  development  effort. 

Weakness 

Activity  performed  3 

The  project  team  conducts  periodic 
reviews  and  interchanges  with  the 
contractor  team. 

The  project  team  does  have  periodic 
reviews  and  discussions  with  the 
contractor  team  that  are  documented. 

Strength 

Activity  performed  4 

The  actual  cost  and  schedule  of  the 
contractor’s  software  effort  are 
compared  to  planned  budgets  and 
schedules,  and  issues  are  identified. 

The  project  team  does  compare  the  actual 
cost  and  schedule  of  the  contractor’s 
software  engineering  effort  to  planned 
schedules  and  budgets,  identifies  issues, 
and  documents  results. 

Strength 

Activity  performed  5 

The  size,  critical  computer  resources, 
and  technical  activities  associated  with 
the  contractor  team’s  work  products  are 
tracked,  and  issues  are  identified. 

The  project  team  does  not  track  the  size, 
critical  computer  resources,  and  technical 
activities  associated  with  the  contractor 
team’s  work  products  or  identify  issues. 

Weakness 
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Public  and  Indian  Housing  Information  Center  system 

Common  feature 

Key  practice 

Finding 

Rating 

Activity  performed  6 

The  project  team  reviews  and  tracks 
the  development  of  the  software 
engineering  environment  required  to 
provide  life-cycle  support  for  the 
acquired  software,  and  issues  are 
identified. 

The  project  team  does  review  and  track 
development  of  the  software  engineering 
environment  required  to  provide  life-cycle 
support  for  the  acquired  software,  and 
issues  are  identified  and  documented. 

Strength 

Activity  performed  7 

Any  issues  found  by  the  project  team 
during  contract  tracking  and  oversight 
activities  are  recorded  in  the 
appropriate  corrective  action  system, 
action  is  taken,  and  issues  are  tracked 
to  closure. 

The  project  team  does  not  have  a 
corrective  action  system  to  record  or  track 
action  taken  on  issues  found  by  the  project 
team  during  contract  tracking  and  oversight 
activities. 

Weakness 

Activity  performed  8 

The  project  team  ensures  that  changes 
to  the  software-related  contractual 
requirements  are  coordinated  with 
affected  groups  and  individuals,  such 
as  the  contracting  official,  contractor, 
and  end  user. 

The  project  team  does  ensure  that 
changes  to  contractual  requirements  are 
coordinated  with  all  affected  groups  and 
individuals. 

Strength 

Measurement  and  analysis  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  contract 
tracking  and  oversight  activities  and 
resultant  products. 

No  measurements  are  made  of  HUD’s 
contract  tracking  and  oversight  activities  or 
resultant  products. 

Weakness 

Verifying  implementation  1 

Contract  tracking  and  oversight 
activities  are  reviewed  by  acquisition 
management  on  a  periodic  basis. 

The  acquisition  organization  does  not 
review  contract  tracking  and  oversight 
activities  on  a  periodic  basis. 

Weakness 

Verifying  implementation  2 

Contract  tracking  and  oversight 
activities  are  reviewed  by  the  project 
manager  on  both  a  periodic  and  an 
event-driven  basis. 

The  project  manager  stated  that  while  he 
discusses  the  status  of  the  project  with 
team  members,  meeting  results  and  action 
items  are  not  documented. 

Weakness 
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Table  14:  Contract  Tracking  and  Oversight  Findings  for  the  Real  Estate  Management  System 

Real  Estate  Management  System 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  to  perform  1 

The  acquisition  organization  has  a 
written  policy  for  managing  the 
contract  tracking  and  oversight  of  the 
software  acquisition  project. 

HUD  has  no  written  policy  for  contract 
tracking  and  oversight  of  the  acquisition  of 
software  products  and  services.  The 
project  team  is  following  the  policy  in 

HUD’s  procurement  manual  for  contract 
tracking  and  oversight. 

Strength 

Commitment  to  perform  2 

Responsibility  has  been  designated  for 
contract  tracking  and  oversight 
activities. 

Responsibility  for  contract  tracking  and 
oversight  activities  has  been  designated  to 
contract  specialists  on  the  team. 

Strength 

Commitment  to  perform  3 

The  project  team  includes  contracting 
specialists  in  the  management  of  the 
contract. 

The  project  team  includes  contract 
specialists  to  help  manage  the  contract. 

Strength 

Ability  to  perform  1 

A  group  is  responsible  for  managing 
contract  tracking  and  oversight 
activities. 

A  group  is  assigned  responsibility  for 
contract  tracking  and  oversight  activities. 

Strength 

Ability  to  perform  2 

Adequate  resources  are  provided  for 
contract  tracking  and  oversight 
activities. 

There  is  no  mechanism  to  identify  resource 
needs  and  ensure  that  they  are  provided  to 
the  project.  Team  members  indicated  that 
the  resources  for  performing  contract 
tracking  and  oversight  activities  are  not 
adequate. 

Weakness 

Ability  to  perform  3 

Individuals  performing  contract 
tracking  and  oversight  activities  have 
experience  or  receive  training. 

Project  team  members  performing  contract 
tracking  and  oversight  activities  have 
experience  and  have  received  training. 

Strength 

Activity  performed  1 

The  project  team  performs  its  activities 
in  accordance  with  its  documented 
contract  tracking  and  oversight  plans. 

There  is  no  contract  tracking  and  oversight 
plan;  therefore,  the  activities  are  not 
performed  in  accordance  with  it. 

Weakness 

Activity  performed  2 

The  project  team  reviews  required 
contractor  software  planning 
documents  which,  when  satisfactory, 
are  used  to  oversee  the  contractor 
team’s  software  effort. 

The  contractor  did  prepare  software 
planning  documents  that  could  be  used  to 
oversee  the  contractor  team’s  software 
development  effort,  but  no  evidence  was 
provided  that  these  were  reviewed  and/or 
approved  by  the  project  team. 

Weakness 

Activity  performed  3 

The  project  team  conducts  periodic 
reviews  and  interchanges  with  the 
contractor  team. 

The  project  team  does  have  periodic 
reviews  and  discussions  with  the 
contractor  team  that  are  documented. 

Strength 

Activity  performed  4 

The  actual  cost  and  schedule  of  the 
contractor’s  software  effort  are 
compared  to  planned  budgets  and 
schedules,  and  issues  are  identified. 

The  project  team  does  compare  the  actual 
cost  and  schedule  of  the  contractor’s 
software  engineering  effort  to  planned 
schedules  and  budgets,  identify  issues, 
and  document  results. 

Strength 

Activity  performed  5 

The  size,  critical  computer  resources, 
and  technical  activities  associated  with 
the  contractor  team’s  work  products 
are  tracked,  and  issues  are  identified. 

The  project  team  told  us  they  track  the 
size,  critical  computer  resources,  and 
technical  activities  associated  with  the 
contractor  team’s  work  products,  but  no 
documentation  was  provided  to  support 
this  statement. 

Weakness 
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Real  Estate  Management  System 

Common  feature 

Key  practice 

Finding 

Rating 

Activity  performed  6 

The  project  team  reviews  and  tracks 
the  development  of  the  software 
engineering  environment  required  to 
provide  life-cycle  support  for  the 
acquired  software,  and  issues  are 
identified. 

The  project  team  does  review  and  track  the 
development  of  the  software  engineering 
environment  required  to  provide  life-cycle 
support  for  the  acquired  software,  and 
issues  are  identified  and  documented. 

Strength 

Activity  performed  7 

Any  issues  found  by  the  project  team 
during  contract  tracking  and  oversight 
activities  are  recorded  in  the 
appropriate  corrective  action  system, 
action  is  taken,  and  issues  are  tracked 
to  closure. 

The  project  team  does  not  have  a 
corrective  action  system  to  record  or  track 
action  taken  on  issues  found  by  the  project 
team  during  contract  tracking  and  oversight 
activities. 

Weakness 

Activity  performed  8 

The  project  team  ensures  that 
changes  to  the  software-related 
contractual  requirements  are 
coordinated  with  affected  groups  and 
individuals,  such  as  the  contracting 
official,  contractor,  and  end  user. 

The  project  team  stated  that  no  changes  to 
the  software-related  contractual 
requirements  have  taken  place  because 
their  contracts  are  broad  enough  to  cover 
changes  to  the  contractual  requirements. 

Not  rated 

Measurement  and  analysis  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  contract 
tracking  and  oversight  activities  and 
resultant  products. 

No  measurements  are  made  of  HUD’s 
contract  tracking  and  oversight  activities  or 
resultant  products. 

Weakness 

Verifying  implementation  1 

Contract  tracking  and  oversight 
activities  are  reviewed  by  acquisition 
management  on  a  periodic  basis. 

The  acquisition  organization  does  not 
review  contract  tracking  and  oversight 
activities  on  a  periodic  basis. 

Weakness 

Verifying  implementation  2 

Contract  tracking  and  oversight 
activities  are  reviewed  by  the  project 
manager  on  both  a  periodic  and  an 
event-driven  basis. 

The  project  manager  informally  reviews  the 
work  of  the  project  team;  however,  the 
results  of  these  reviews  are  not 
documented.  Minutes  of  biweekly  status 
meetings  with  contractors  are  maintained. 

Observation 
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Table  15:  Contract  Tracking  and  Oversight  Findings  for  the  Resident  Assessment  Subsystem 

Resident  Assessment  Subsystem 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  to  perform  1 

The  acquisition  organization  has  a 
written  policy  for  managing  the 
contract  tracking  and  oversight  of  the 
software  acquisition  project. 

HUD  has  no  written  policy  for  contract 
tracking  and  oversight  of  the  acquisition  of 
software  products  and  services.  The 
project  team  is  following  the  policy  in 

HUD’s  Procurement  Handbook  for  contract 
tracking  and  oversight. 

Strength 

Commitment  to  perform  2 

Responsibility  has  been  designated  for 
contract  tracking  and  oversight 
activities. 

Responsibility  for  contract  tracking  and 
oversight  activities  is  designated  in  the 
project  plan. 

Strength 

Commitment  to  perform  3 

The  project  team  includes  contracting 
specialists  in  the  management  of  the 
contract. 

The  project  team  includes  contract 
specialists  to  help  manage  the  contract. 

Strength 

Ability  to  perform  1 

A  group  is  responsible  for  managing 
contract  tracking  and  oversight 
activities. 

A  group  is  assigned  responsibility  for 
contract  tracking  and  oversight  activities. 

Strength 

Ability  to  perform  2 

Adequate  resources  are  provided  for 
contract  tracking  and  oversight 
activities. 

There  is  no  mechanism  to  identify  resource 
needs  and  ensure  that  they  are  provided  to 
the  project.  The  project  manager  stated 
that  the  team  does  have  adequate 
resources  for  performing  contract  tracking 
and  oversight  activities. 

Strength 

Ability  to  perform  3 

Individuals  performing  contract 
tracking  and  oversight  activities  have 
experience  or  receive  training. 

Project  team  members  performing  contract 
tracking  and  oversight  activities  have 
experience  and  have  received  training. 

Strength 

Activity  performed  1 

The  project  team  performs  its  activities 
in  accordance  with  its  documented 
contract  tracking  and  oversight  plans. 

The  project  team  performs  its  activities  in 
accordance  with  its  documented  contract 
tracking  and  oversight  plan. 

Strength 

Activity  performed  2 

The  project  team  reviews  required 
contractor  software  planning 
documents  which,  when  satisfactory, 
are  used  to  oversee  the  contractor 
team’s  software  effort. 

The  contractor  did  not  prepare  software 
planning  documents  that  the  project  team 
could  use  to  oversee  the  contractor’s 
software  development  effort. 

Weakness 

Activity  performed  3 

The  project  team  conducts  periodic 
reviews  and  interchanges  with  the 
contractor  team. 

The  project  team  does  have  periodic 
reviews  and  discussions  with  the 
contractor  team  that  are  documented. 

Strength 

Activity  performed  4 

The  actual  cost  and  schedule  of  the 
contractor’s  software  effort  are 
compared  to  planned  budgets  and 
schedules,  and  issues  are  identified. 

The  project  team  does  compare  the  actual 
cost  and  schedule  of  the  contractor’s 
software  engineering  effort  to  planned 
schedules  and  budgets,  identify  issues, 
and  document  results. 

Strength 

Activity  performed  5 

The  size,  critical  computer  resources, 
and  technical  activities  associated  with 
the  contractor  team’s  work  products 
are  tracked,  and  issues  are  identified. 

The  project  team  does  track  the  size, 
critical  computer  resources,  and  technical 
activities  associated  with  the  contractor 
team’s  work  products  and  identifies  and 
documents  issues. 

Strength 
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Resident  Assessment  Subsystem 

Common  feature 

Key  practice 

Finding 

Rating 

Activity  performed  6 

The  project  team  reviews  and  tracks 
the  development  of  the  software 
engineering  environment  required  to 
provide  life-cycle  support  for  the 
acquired  software,  and  issues  are 
identified. 

The  project  team  does  review  and  track  the 
development  of  the  software  engineering 
environment  required  to  provide  life-cycle 
support  for  the  acquired  software,  and 
issues  are  identified  and  documented. 

Strength 

Activity  performed  7 

Any  issues  found  by  the  project  team 
during  contract  tracking  and  oversight 
activities  are  recorded  in  the 
appropriate  corrective  action  system, 
action  is  taken,  and  issues  are  tracked 
to  closure. 

The  project  team  records  issues  related  to 
contract  tracking  and  oversight  in  a 
corrective  action  system,  and  tracks 
corrective  actions  to  closure. 

Strength 

Activity  performed  8 

The  project  team  ensures  that 
changes  to  the  software-related 
contractual  requirements  are 
coordinated  with  affected  groups  and 
individuals,  such  as  the  contracting 
official,  contractor,  and  end  user. 

The  project  team  stated  that  no  changes  to 
the  software-related  contractual 
requirements  have  taken  place. 

Not  rated 

Measurement  and  analysis  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  contract 
tracking  and  oversight  activities  and 
resultant  products. 

No  measurements  are  made  of  HUD’s 
contract  tracking  and  oversight  activities 
and  resultant  products. 

Weakness 

Verifying  implementation  1 

Contract  tracking  and  oversight 
activities  are  reviewed  by  acquisition 
management  on  a  periodic  basis. 

The  acquisition  organization  does  not 
review  contract  tracking  and  oversight 
activities  on  a  periodic  basis. 

Weakness 

Verifying  implementation  2 

Contract  tracking  and  oversight 
activities  are  reviewed  by  the  project 
manager  on  both  a  periodic  and  an 
event-driven  basis. 

The  project  manager  informally  reviews  the 
work  of  the  project  team.  However,  the 
results  of  these  reviews  are  not 
documented. 

Weakness 
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Table  16:  Contract  Tracking  and  Oversight  Findings  for  HUD’s  Central  Accounting  and  Program  System 

HUD’s  Central  Accounting  and  Program  System 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  to  perform  1 

The  acquisition  organization  has  a 
written  policy  for  managing  the 
contract  tracking  and  oversight  of  the 
software  acquisition  project. 

HUD  has  no  written  policy  for  contract 
tracking  and  oversight  of  the  acquisition  of 
software  products  and  services.  The 
project  team  is  following  the  policy  in 

HUD’s  Procurement  Handbook  for  contract 
tracking  and  oversight. 

Strength 

Commitment  to  perform  2 

Responsibility  has  been  designated  for 
contract  tracking  and  oversight 
activities. 

Responsibility  for  contract  tracking  and 
oversight  activities  is  designated  in  the 
organization  chart. 

Strength 

Commitment  to  perform  3 

The  project  team  includes  contracting 
specialists  in  the  management  of  the 
contract. 

The  project  team  includes  contract 
specialists  to  help  manage  the  contract. 

Strength 

Ability  to  perform  1 

A  group  is  responsible  for  managing 
contract  tracking  and  oversight 
activities. 

A  group  is  assigned  responsibility  for 
contract  tracking  and  oversight  activities. 

Strength 

Ability  to  perform  2 

Adequate  resources  are  provided  for 
contract  tracking  and  oversight 
activities. 

There  is  no  mechanism  to  identify  resource 
needs  and  ensure  that  they  are  provided  to 
the  project.  The  project  manager  stated 
that  the  team  does  not  have  adequate 
resources  for  performing  contract  tracking 
and  oversight  activities. 

Weakness 

Ability  to  perform  3 

Individuals  performing  contract 
tracking  and  oversight  activities  have 
experience  or  receive  training. 

Project  team  members  performing  contract 
tracking  and  oversight  activities  have 
experience  and  have  received  training. 

Strength 

Activity  performed  1 

The  project  team  performs  its  activities 
in  accordance  with  its  documented 
contract  tracking  and  oversight  plans. 

There  is  no  contract  tracking  and  oversight 
plan;  therefore,  the  activities  are  not 
performed  in  accordance  with  it. 

Weakness 

Activity  performed  2 

The  project  team  reviews  required 
contractor  software  planning 
documents  which,  when  satisfactory, 
are  used  to  oversee  the  contractor 
team’s  software  effort. 

The  contractor  did  not  prepare  software 
planning  documents  that  the  project  team 
could  use  to  oversee  the  contractor’s 
software  development  effort. 

Weakness 

Activity  performed  3 

The  project  team  conducts  periodic 
reviews  and  interchanges  with  the 
contractor  team. 

The  project  team  does  have  periodic 
reviews  and  discussions  with  the 
contractor  team  that  are  documented. 

Strength 

Activity  performed  4 

The  actual  cost  and  schedule  of  the 
contractor’s  software  effort  are 
compared  to  planned  budgets  and 
schedules,  and  issues  are  identified. 

The  project  team  told  us  they  do  compare 
the  actual  cost  and  schedule  of  the 
contractor’s  software  engineering  effort  to 
planned  schedules  and  budgets,  but  no 
documentation  was  provided  to  support 
this  statement. 

Observation 

Activity  performed  5 

The  size,  critical  computer  resources, 
and  technical  activities  associated  with 
the  contractor  team’s  work  products 
are  tracked,  and  issues  are  identified. 

The  project  team  does  track  the  size, 
critical  computer  resources,  and  technical 
activities  associated  with  the  contractor 
team’s  work  products  and  identifies  and 
documents  issues. 

Strength 
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HUD’s  Central  Accounting  and  Program  System 

Common  feature 

Key  practice 

Finding 

Rating 

Activity  performed  6 

The  project  team  reviews  and  tracks 
the  development  of  the  software 
engineering  environment  required  to 
provide  life-cycle  support  for  the 
acquired  software,  and  issues  are 
identified. 

The  project  team  does  review  and  track  the 
development  of  the  software  engineering 
environment  required  to  provide  life-cycle 
support  for  the  acquired  software,  and 
issues  are  identified  and  documented. 

Strength 

Activity  performed  7 

Any  issues  found  by  the  project  team 
during  contract  tracking  and  oversight 
activities  are  recorded  in  the 
appropriate  corrective  action  system, 
action  is  taken,  and  issues  are  tracked 
to  closure. 

The  project  team  does  not  have  a 
corrective  action  system  to  record  or  track 
action  taken  on  issues  found  by  the  project 
team  during  contract  tracking  and  oversight 
activities. 

Weakness 

Activity  performed  8 

The  project  team  ensures  that 
changes  to  the  software-related 
contractual  requirements  are 
coordinated  with  affected  groups  and 
individuals,  such  as  the  contracting 
official,  contractor,  and  end  user. 

The  project  team  stated  that  no  changes  to 
the  software-related  contractual 
requirements  have  taken  place  because 
their  contracts  are  broad  enough  to  cover 
changes  to  contractual  requirements. 

Not  rated 

Measurement  and  analysis  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  contract 
tracking  and  oversight  activities  and 
resultant  products. 

No  measurements  are  made  of  HUD’s 
contract  tracking  and  oversight  activities  or 
resultant  products. 

Weakness 

Verifying  implementation  1 

Contract  tracking  and  oversight 
activities  are  reviewed  by  acquisition 
management  on  a  periodic  basis. 

The  acquisition  organization  regularly 
reviews  contract  tracking  and  oversight 
activities. 

Strength 

Verifying  implementation  2 

Contract  tracking  and  oversight 
activities  are  reviewed  by  the  project 
manager  on  both  a  periodic  and  an 
event-driven  basis. 

The  project  manager  is  briefed  weekly  on 
project  status,  including  contract  tracking 
and  oversight  activities. 

Strength 
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Table  17:  Contract  Tracking  and  Oversight  Findings  for  the  Empowerment  Information  System 


Empowerment  Information  System 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  to  perform  1 

The  acquisition  organization  has  a 
written  policy  for  managing  the 
contract  tracking  and  oversight  of  the 
software  acquisition  project. 

HUD  has  no  written  policy  for  contract 
tracking  and  oversight  of  the  acquisition  of 
software  products  and  services.  The 
project  team  is  following  the  policy  in 

HUD’s  Procurement  Handbook  for 
contract  tracking  and  oversight. 

Strength 

Commitment  to  perform  2 

Responsibility  has  been  designated  for 
contract  tracking  and  oversight 
activities. 

Responsibility  for  contract  tracking  and 
oversight  activities  has  been  designated 
in  the  organization  chart. 

Strength 

Commitment  to  perform  3 

The  project  team  includes  contracting 
specialists  in  the  management  of  the 
contract. 

The  project  team  includes  contract 
specialists  to  help  manage  the  contract. 

Strength 

Ability  to  perform  1 

A  group  is  responsible  for  managing 
contract  tracking  and  oversight 
activities. 

A  group  is  assigned  responsibility  for 
managing  contract  tracking  and  oversight 
activities. 

Strength 

Ability  to  perform  2 

Adequate  resources  are  provided  for 
contract  tracking  and  oversight 
activities. 

There  is  no  mechanism  to  identify 
resource  needs  and  ensure  that  they  are 
provided  to  the  project.  The  project 
manager  indicated  that  the  team  does  not 
have  adequate  resources  for  performing 
contract  tracking  and  oversight  activities. 

Weakness 

Ability  to  perform  3 

Individuals  performing  contract 
tracking  and  oversight  activities  have 
experience  or  receive  training. 

Project  team  members  performing 
contract  tracking  and  oversight  activities 
have  experience  and  have  received 
training. 

Strength 

Activity  performed  1 

The  project  team  performs  its  activities 
in  accordance  with  its  documented 
contract  tracking  and  oversight  plans. 

There  is  no  contract  tracking  and 
oversight  plan;  therefore,  the  activities  are 
not  performed  in  accordance  with  it. 

Weakness 

Activity  performed  2 

The  project  team  reviews  required 
contractor  software  planning 
documents  which,  when  satisfactory, 
are  used  to  oversee  the  contractor 
team’s  software  effort. 

The  contractor  did  not  prepare  software 
planning  documents  that  the  project  team 
could  use  to  oversee  the  contractor’s 
software  development  effort. 

Weakness 

Activity  performed  3 

The  project  team  conducts  periodic 
reviews  and  interchanges  with  the 
contractor  team. 

The  project  team  does  have  periodic 
reviews  and  discussions  with  the 
contractor  team  that  are  documented. 

Strength 

Activity  performed  4 

The  actual  cost  and  schedule  of  the 
contractor’s  software  effort  are 
compared  to  planned  budgets  and 
schedules,  and  issues  are  identified. 

The  project  team  does  compare  the 
actual  cost  and  schedule  of  the 
contractor’s  software  engineering  effort  to 
planned  schedules  and  budgets,  identify 
issues,  and  document  results. 

Strength 

Activity  performed  5 

The  size,  critical  computer  resources, 
and  technical  activities  associated  with 
the  contractor  team’s  work  products 
are  tracked,  and  issues  are  identified. 

The  project  team  reviewed  the  size, 
critical  computer  resources,  and  technical 
activities  associated  with  the  contractor 
team’s  work  products  only  during  pre¬ 
release  testing. 

Observation 
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Empowerment  Information  System 

Common  feature 

Key  practice 

Finding 

Rating 

Activity  performed  6 

The  project  team  reviews  and  tracks 
the  development  of  the  software 
engineering  environment  required  to 
provide  life-cycle  support  for  the 
acquired  software,  and  issues  are 
identified. 

The  project  does  review  and  track 
development  of  the  software  engineering 
environment  required  to  provide  life-cycle 
support  for  the  acquired  software,  and 
issues  are  identified  and  documented. 

Strength 

Activity  performed  7 

Any  issues  found  by  the  project  team 
during  contract  tracking  and  oversight 
activities  are  recorded  in  the 
appropriate  corrective  action  system, 
action  is  taken,  and  issues  are  tracked 
to  closure. 

The  project  team  records  issues  related  to 
contract  tracking  and  oversight  in  a 
corrective  action  system,  and  tracks 
corrective  actions  to  closure. 

Strength 

Activity  performed  8 

The  project  team  ensures  that 
changes  to  the  software-related 
contractual  requirements  are 
coordinated  with  affected  groups  and 
individuals,  such  as  the  contracting 
official,  contractor,  and  end  user. 

The  project  team  stated  that  changes  to 
software-related  contractual  requirements 
were  coordinated  with  all  affected  groups 
and  individuals,  but  they  provided  no 
documentation  to  support  this  statement. 

Weakness 

Measurement  and  analysis  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  contract 
tracking  and  oversight  activities  and 
resultant  products. 

No  measurements  are  made  of  HUD’s 
contract  tracking  and  oversight  activities 
or  resultant  products. 

Weakness 

Verifying  implementation  1 

Contract  tracking  and  oversight 
activities  are  reviewed  by  acquisition 
management  on  a  periodic  basis. 

The  acquisition  organization  periodically 
reviews  contract  tracking  and  oversight 
activities. 

Strength 

Verifying  implementation  2 

Contract  tracking  and  oversight 
activities  are  reviewed  by  the  project 
manager  on  both  a  periodic  and  an 
event-driven  basis. 

The  project  manager  works  closely  with 
and  informally  reviews  the  work  of  the 
project  team.  However,  the  results  of 
these  reviews  are  not  documented. 

Observation 

Page  53 


GAO-Ol-962  HUD  Information  Systems 


Chapter  5:  HUD’s  Software  Evaluation 
Processes  Are  Not  Repeatable 


HUD  did  not  satisfy  the  criteria  for  the  repeatable  level  of  maturity  in  this 
key  process  area  because  of  the  number  and  severity  of  key  practice 
weaknesses  found.  These  weaknesses  result  in  increased  risk  that  HUD’s 
software  acquisition  projects  will  not  be  evaluated  effectively,  software 
products  will  not  work  effectively  or  meet  mission  needs,  and  cost, 
schedule,  and  other  performance  goals  will  not  be  met. 

The  purpose  of  evaluation  is  to  determine  that  the  acquired  software 
products  and  services  satisfy  contract  requirements  before  acceptance. 
Within  this  key  process  area,  the  SA-CMM  specifies  key  practices  for  each 
of  the  five  common  features  that  an  organization  must  implement 
effectively  to  achieve  the  repeatable  level  of  maturity.  Generally,  these 
practices  include  having  a  written  organizational  policy  for  evaluating 
acquired  software  products  and  services,  documenting  plans  for 
evaluating  software  products  and  services,  developing  and  managing 
evaluation  requirements  in  conjunction  with  software  technical 
requirements,  tracking  contractor  evaluation  activities  for  compliance 
with  the  contract,  and  measuring  and  reporting  on  the  status  and  results  of 
evaluation  activities  to  management. 

For  the  75  key  practices  in  this  key  process  area,  HUD  had  37  strengths,  33 
weaknesses,  and  five  observations.  For  the 

•  Commitment  to  perform  feature,  there  were  four  strengths,  four 
weaknesses,  and  two  observations. 

•  Ability  to  perform  feature,  there  were  1 1  strengths  and  nine  weaknesses. 

•  Activities  performed  feature,  there  were  19  strengths,  nine  weaknesses, 
and  two  observations. 

•  Measurement  and  analysis  feature,  there  were  no  strengths,  four 
weaknesses,  and  one  observation. 

•  Verifying  implementation  feature,  there  were  three  strengths  and  seven 
weaknesses. 

In  addition,  none  of  the  projects  had  strengths  in  all  the  key  practices. 

As  a  result  of  these  weaknesses,  HUD  is  exposed  to  increased  risk  that 
software  acquisition  projects  will  not  be  evaluated  effectively,  that 
acquired  software  will  not  work  efficiently  or  meet  mission  needs,  and  that 
cost,  schedule,  and  other  performance  goals  will  not  be  met.  HUD 
acknowledges  the  need  to  improve  its  software  evaluation  processes  and 
is  committed  to  improving  its  software  and  system  acquisition  capability. 
The  department  has  prepared  a  statement  of  work  to  obtain  contract 
support  to  begin  a  software  process  improvement  effort.  This  document 
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specifies  that  the  contractor  would  review  HUD  software  acquisition 
processes,  prioritize  the  key  processes  and  practices  to  address,  and 
develop  a  plan  to  improve  HUD’s  acquisition  processes. 

Details  of  our  evaluation  are  provided  in  the  following  figure  and  tables. 
Figure  5  provides  a  comprehensive  listing  of  the  strengths,  weaknesses, 
and  observations  in  the  evaluation  key  process  area.  The  specific  findings 
supporting  the  practice  ratings  cited  in  figure  5  are  in  tables  18  through  22. 
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Figure  5:  Software  Evaluation  Summary 


Projects 

features 

Key  practices 

PIC 

REMS 

RASS 

HUDCAPS 

EIS 

Commitment  to 
perform  1 

The  acquisition  organization  has  a  written  policy  for 
managing  the  evaluation  of  acquired  software  products 
and  services. 

Weakness 

Weakness 

Observation 

Commitment  to 

Responsibility  has  been  designated  for  evaluation 

Weakness 

Strength 

Strength 

Strength 

Strength 

perform  2 

activities. 

Ability  to 
perform  1 

A  group  is  responsible  for  planning,  managing,  and 
performing  evaluation  activities. 

Weakness 

Strength 

Strength 

Strength 

Strength 

Ability  to 
perform  2 

Adequate  resources  are  provided  for  evaluation 
activities. 

Weakness 

Weakness 

Strength 

Weakness 

Weakness 

Ability  to 
perform  3 

Individuals  performing  evaluation  activities  have 
experience  or  receive  training. 

Strength 

Strength 

Strength 

Strength 

Strength 

Ability  to 
perform  4 

Members  of  the  project  team  and  groups  supporting  the 
software  acquisition  receive  orientation  on  the 
objectives  of  the  evaluation  approach. 

Weakness 

Weakness 

Weakness 

Weakness 

Strength 

Activity 
performed  1 

The  project  team  performs  its  activities  in  accordance 
with  its  documented  evaluation  plans. 

Strength 

Strength 

Strength 

Weakness 

Weakness 

Activity 
performed  2 

The  project’s  evaluation  requirements  are  developed  in 
conjunction  with  the  development  of  the  system  or 
software  technical  requirements. 

Strength 

Observation 

Strength 

Strength 

Weakness 

Activity 
performed  3 

The  project’s  evaluation  activities  are  planned  to 
minimize  duplication  and  take  advantage  of  all 
evaluation  results,  where  appropriate. 

Strength 

Strength 

Strength 

Strength 

Weakness 

Activity 
performed  4 

The  project  team  appraises  the  contractor  team’s 
performance  over  the  total  period  of  the  contract  for 
compliance  with  requirements. 

Weakness 

Observation 

Strength 

Strength 

Activity 
performed  5 

Planned  evaluations  are  performed  on  the  evolving 
software  products  and  services  before  acceptance  for 
operational  use. 

Strength 

Strength 

Strength 

Strength 

Activity 
performed  6 

Results  of  the  evaluations  are  analyzed  and  compared 
to  the  contract’s  requirements  to  establish  an  objective 
basis  to  support  the  decision  to  accept  the  product  or 
services  or  to  take  further  action. 

Weakness 

Strength 

Strength 

Strength 

Measurement 
and  analysis  1 

Measurements  are  made  and  used  to  determine  the 
status  of  the  evaluation  activities  and  resultant  products. 

Weakness 

Weakness 

Weakness. 

Weakness 

Observation 

Verifying 
implementation  1 

Evaluation  activities  are  reviewed  by  acquisition 
organization  management  on  a  periodic  basis. 

Strength 

Strength 

Weakness 

Weakness 

Weakness 

Verifying 
implementation  2 

Evaluation  activities  are  reviewed  by  the  project 
manager  on  both  a  periodic  and  an  event-driven  basis. 

Weakness 

Strength 

Weakness 

Weakness 

Weakness 

Key 


Strength 


Weakness 

Observation 


Strength:  Key  practice  effectively  implemented. 

Weakness:  Key  practice  not  effectively  implemented. 

Observation:  Key  practice  evaluated;  evidence  not  sufficient  to  rate  as  a  strength,  or  practice  only  partially  performed. 
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Table  18:  Software  Evaluation  Findings  for  the  Public  and  Indian  Housing  Information  Center  System 

Public  and  Indian  Housing  Information  Center  system 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  to  perform  1 

The  acquisition  organization  has  a 
written  policy  for  managing  the 
evaluation  of  acquired  software 
products  and  services. 

HUD  has  no  written  policy  for  managing  the 
evaluation  of  acquired  software  products 
and  services. 

Weakness 

Commitment  to  perform  2 

Responsibility  has  been  designated 
for  evaluation  activities. 

The  project  team  told  us  that  responsibility 
for  evaluation  activities  has  been  assigned 
to  the  lead  technical  person,  but  no 
documentation  was  provided  to  support  this 
statement. 

Weakness 

Ability  to  perform  1 

A  group  is  responsible  for  planning, 
managing,  and  performing  evaluation 
activities. 

The  project  team  told  us  that  evaluation 
activities  are  managed  by  the  lead  technical 
person,  but  no  documentation  was  provided 
to  support  this  statement. 

Weakness 

Ability  to  perform  2 

Adequate  resources  are  provided  for 
evaluation  activities. 

There  is  no  mechanism  to  identify  needed 
resources  and  ensure  that  they  are  provided 
to  the  project.  The  lead  technical  person 
indicated  that  the  team  does  not  have 
adequate  resources  for  performing 
evaluation  activities. 

Weakness 

Ability  to  perform  3 

Individuals  performing  evaluation 
activities  have  experience  or  receive 
training. 

Project  team  members  performing 
evaluation  activities  have  experience. 

Strength 

Ability  to  perform  4 

Members  of  the  project  team  and 
groups  supporting  the  software 
acquisition  receive  orientation  on  the 
objectives  of  the  evaluation 
approach. 

Members  of  the  project  team  or  others 
supporting  the  software  acquisition  did  not 
receive  orientation  on  the  objectives  of  the 
evaluation  approach. 

Weakness 

Activity  performed  1 

The  project  team  performs  its 
activities  in  accordance  with  its 
documented  evaluation  plans. 

The  project  team  performs  its  evaluation 
activities  in  accordance  with  its  documented 
evaluation  plans. 

Strength 

Activity  performed  2 

The  project’s  evaluation  requirements 
are  developed  in  conjunction  with  the 
development  of  the  system  or 
software  technical  requirements. 

Project  evaluation  requirements  were 
developed  in  conjunction  with  software 
technical  requirements  and  are  designated 
in  the  test  plans  and  test  scripts. 

Strength 

Activity  performed  3 

The  project’s  evaluation  activities  are 
planned  to  minimize  duplication  and 
take  advantage  of  all  evaluation 
results,  where  appropriate. 

There  appears  to  be  no  duplication  of 
evaluation  activities. 

Strength 

Activity  performed  4 

The  project  team  appraises  the 
contractor  team’s  performance  over 
the  total  period  of  the  contract  for 
compliance  with  requirements. 

The  project  team  stated  that  quarterly 
reports  of  contractor  deliverables  were  used 
to  appraise  contractor  performance,  but  no 
documentation  was  provided  to  support  this 
statement. 

Weakness 

Activity  performed  5 

Planned  evaluations  are  performed 
on  the  evolving  software  products 
and  services  before  acceptance  for 
operational  use. 

Several  evaluations  are  performed  on 
evolving  software  products  and  services 
before  acceptance  for  operational  use. 

Strength 
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Public  and  Indian  Housing  Information  Center  system 

Common  feature 

Key  practice 

Finding 

Rating 

Activity  performed  6 

Results  of  the  evaluations  are 
analyzed  and  compared  to  the 
contract’s  requirements  to  establish 
an  objective  basis  to  support  the 
decision  to  accept  the  product  or 
services  or  to  take  further  action. 

There  is  no  analysis  or  comparison  of 
evaluation  results  to  contract  requirements 
to  establish  an  objective  basis  supporting 
the  decision  to  accept  the  products  and 
services  or  to  take  further  action. 

Weakness 

Measurement  and  analysis  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  evaluation 
activities  and  resultant  products. 

No  measurements  are  made  of  HUD’s 
evaluation  activities  and  resultant  products. 

Weakness 

Verifying  implementation  1 

Evaluation  activities  are  reviewed  by 
acquisition  organization  management 
on  a  periodic  basis. 

Acquisition  organization  management 
regularly  reviews  evaluation  activities. 

Strength 

Verifying  implementation  2 

Evaluation  activities  are  reviewed  by 
the  project  manager  on  both  a 
periodic  and  an  event-driven  basis. 

The  project  manager  does  not  formally 
review  evaluation  activities,  and  the  project 
manager’s  reviews  are  not  documented. 

Weakness 
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Table  19:  Software  Evaluation  Findings  for  the  Real  Estate  Management  System 

Real  Estate  Management  System 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  to  perform  1 

The  acquisition  organization  has  a 
written  policy  for  managing  the 
evaluation  of  acquired  software 
products  and  services. 

HUD  has  no  written  policy  for  managing  the 
evaluation  of  acquired  software  products 
and  services. 

Weakness 

Commitment  to  perform  2 

Responsibility  has  been  designated  for 
evaluation  activities. 

Contractors  and  users  have  been  assigned 
responsibility  for  evaluation  activities. 

Strength 

Ability  to  perform  1 

A  group  is  responsible  for  planning, 
managing,  and  performing  evaluation 
activities. 

The  contractor  and  end  users  are 
responsible  for  evaluation  activities. 

Strength 

Ability  to  perform  2 

Adequate  resources  are  provided  for 
evaluation  activities. 

There  is  no  mechanism  to  identify  needed 
resources  and  ensure  that  they  are  provided 
to  the  project.  The  project  manager  stated 
that  the  team  does  not  have  adequate 
resources  for  performing  evaluation 
activities. 

Weakness 

Ability  to  perform  3 

Individuals  performing  evaluation 
activities  have  experience  or  receive 
training. 

Project  team  members  performing 
evaluation  activities  have  experience. 

Strength 

Ability  to  perform  4 

Members  of  the  project  team  and 
groups  supporting  the  software 
acquisition  receive  orientation  on  the 
objectives  of  the  evaluation  approach. 

Members  of  the  project  team  or  others 
supporting  the  software  acquisition  did  not 
receive  orientation  on  the  objectives  of  the 
evaluation  approach. 

Weakness 

Activity  performed  1 

The  project  team  performs  its  activities 
in  accordance  with  its  documented 
evaluation  plans. 

The  team  performs  its  evaluation  activities  in 
accordance  with  its  documented  evaluation 
plans. 

Strength 

Activity  performed  2 

The  project’s  evaluation  requirements 
are  developed  in  conjunction  with  the 
development  of  the  system  or  software 
technical  requirements. 

Project  evaluation  requirements  are 
documented  in  the  project  plan,  but  no 
documentation  was  provided  showing  that 
they  were  developed  in  conjunction  with 
development  of  the  software  technical 
requirements. 

Observation 

Activity  performed  3 

The  project’s  evaluation  activities  are 
planned  to  minimize  duplication  and 
take  advantage  of  all  evaluation  results, 
where  appropriate. 

There  appears  to  be  no  duplication  of 
evaluation  activities. 

Strength 

Activity  performed  4 

The  project  team  appraises  the 
contractor  team’s  performance  over  the 
total  period  of  the  contract  for 
compliance  with  requirements. 

The  project  team  assesses  contractor  team 
performance  for  compliance  with 
requirements  only  during  user  acceptance 
testing. 

Observation 

Activity  performed  5 

Planned  evaluations  are  performed  on 
the  evolving  software  products  and 
services  before  acceptance  for 
operational  use. 

Planned  evaluations  are  performed  before 
software  releases  are  accepted  for 
operational  use. 

Strength 
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Real  Estate  Management  System 

Common  feature 

Key  practice 

Finding 

Rating 

Activity  performed  6 

Results  of  the  evaluations  are  analyzed 
and  compared  to  the  contract’s 
requirements  to  establish  an  objective 
basis  to  support  the  decision  to  accept 
the  product  or  services  or  to  take 
further  action. 

The  results  of  evaluations  are  analyzed  and 
compared  to  the  contract’s  requirements  to 
establish  an  objective  basis  supporting  the 
decision  to  accept  the  products  and  services 
or  to  take  further  action. 

Strength 

Measurement  and  analysis  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  evaluation 
activities  and  resultant  products. 

No  measurements  are  made  of  HUD’s 
evaluation  activities  or  resultant  products. 

Weakness 

Verifying  implementation  1 

Evaluation  activities  are  reviewed  by 
acquisition  organization  management 
on  a  periodic  basis. 

Acquisition  organization  management 
regularly  reviews  evaluation  activities. 

Strength 

Verifying  implementation  2 

Evaluation  activities  are  reviewed  by 
the  project  manager  on  both  a  periodic 
and  an  event-driven  basis. 

Project  reviews  conducted  by  the  project 
manager  cover  evaluation  activities. 

Strength 
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Table  20:  Software  Evaluation  Findings  for  the  Resident  Assessment  Subsystem 

Resident  Assessment  Subsystem 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  to  perform  1 

The  acquisition  organization  has  a 
written  policy  for  managing  the 
evaluation  of  acquired  software 
products  and  services. 

HUD  has  no  written  policy  for  managing  the 
evaluation  of  acquired  software  products  and 
services.  The  project  is  following  the 
evaluation  policies  in  HUD’s  software 
development  methodology. 

Observation 

Commitment  to  perform  2 

Responsibility  has  been  designated 
for  evaluation  activities. 

Responsibility  for  evaluation  activities  has 
been  assigned  to  the  IT  project  lead. 

Strength 

Ability  to  perform  1 

A  group  is  responsible  for  planning, 
managing,  and  performing  evaluation 
activities. 

The  IT  project  lead  and  support  staff  are 
responsible  for  evaluation  activities. 

Strength 

Ability  to  perform  2 

Adequate  resources  are  provided  for 
evaluation  activities. 

There  is  no  mechanism  to  identify  resource 
needs  and  ensure  that  they  are  provided  to 
the  project.  According  to  the  project  manager, 
the  project  team  has  adequate  resources  for 
conducting  evaluation  activities. 

Strength 

Ability  to  perform  3 

Individuals  performing  evaluation 
activities  have  experience  or  receive 
training. 

Project  team  members  performing  evaluation 
activities  have  experience  and  have  received 
training. 

Strength 

Ability  to  perform  4 

Members  of  the  project  team  and 
groups  supporting  the  software 
acquisition  receive  orientation  on  the 
objectives  of  the  evaluation 
approach. 

Members  of  the  project  team  or  others 
supporting  the  software  acquisition  did  not 
receive  orientation  on  the  objectives  of  the 
evaluation  approach. 

Weakness 

Activity  performed  1 

The  project  team  performs  its 
activities  in  accordance  with  its 
documented  evaluation  plans. 

The  project  team  performs  its  evaluation 
activities  in  accordance  with  its  documented 
evaluation  plan. 

Strength 

Activity  performed  2 

The  project’s  evaluation  requirements 
are  developed  in  conjunction  with  the 
development  of  the  system  or 
software  technical  requirements. 

Project  evaluation  requirements  are  developed 
in  conjunction  with  software  technical 
requirements. 

Strength 

Activity  performed  3 

The  project’s  evaluation  activities  are 
planned  to  minimize  duplication  and 
take  advantage  of  all  evaluation 
results,  where  appropriate. 

There  appears  to  be  no  duplication  of 
evaluation  activities. 

Strength 

Activity  performed  4 

The  project  team  appraises  the 
contractor  team’s  performance  over 
the  total  period  of  the  contract  for 
compliance  with  requirements. 

The  project  team  appraises  the  contractor 
team’s  performance  over  the  total  period  of 
the  contract  for  compliance  with  requirements. 

Strength 

Activity  performed  5 

Planned  evaluations  are  performed 
on  the  evolving  software  products 
and  services  before  acceptance  for 
operational  use. 

Planned  evaluation  activities  are  performed  on 
the  evolving  software  products  and  services 
before  they  are  accepted  for  operational  use. 

Strength 

Activity  performed  6 

Results  of  the  evaluations  are 
analyzed  and  compared  to  the 
contract’s  requirements  to  establish 
an  objective  basis  to  support  the 
decision  to  accept  the  product  or 
services  or  to  take  further  action. 

Results  of  the  evaluations  are  analyzed  and 
compared  to  the  contract’s  requirements  to 
establish  an  objective  basis  supporting  the 
decision  to  accept  the  software  products  or  to 
take  further  action. 

Strength 
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Resident  Assessment  Subsystem 

Common  feature 

Key  practice 

Finding 

Rating 

Measurement  and  analysis  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  evaluation 
activities  and  resultant  products. 

No  measurements  are  made  of  HUD’s 
evaluation  activities  and  resultant  products. 

Weakness 

Verifying  implementation  1 

Evaluation  activities  are  reviewed  by 
acquisition  organization  management 
on  a  periodic  basis. 

Acquisition  organization  management  does 
not  review  evaluation  activities  on  a  regular 
basis. 

Weakness 

Verifying  implementation  2 

Evaluation  activities  are  reviewed  by 
the  project  manager  on  both  a 
periodic  and  an  event-driven  basis. 

The  project  manager  stated  that  while  he 
meets  with  his  staff  daily  to  discuss  the  status 
of  the  project,  meeting  results  and  action  items 
are  not  documented. 

Weakness 
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Table  21:  Software  Evaluation  Findings  for  HUD’s  Central  Accounting  and  Program  System 

HUD’s  Central  Accounting  and  Program  System 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  to  perform  1 

The  acquisition  organization  has  a 
written  policy  for  managing  the 
evaluation  of  acquired  software 
products  and  services. 

HUD  has  no  written  policy  for  managing  the 
evaluation  of  acquired  software  products 
and  services.  The  project  is  following  the 
evaluation  policy  in  HUD’s  software 
development  methodology. 

Observation 

Commitment  to  perform  2 

Responsibility  has  been  designated  for 
evaluation  activities. 

Responsibility  for  evaluation  activities  has 
been  assigned  to  the  project  lead. 

Strength 

Ability  to  perform  1 

A  group  is  responsible  for  planning, 
managing,  and  performing  evaluation 
activities. 

The  project  lead  and  support  staff  are 
responsible  for  evaluation  activities. 

Strength 

Ability  to  perform  2 

Adequate  resources  are  provided  for 
evaluation  activities. 

There  is  no  mechanism  to  identify  needed 
resources  and  ensure  that  they  are  provided 
to  the  project.  The  project  manager  stated 
that  the  team  does  not  have  adequate 
resources  for  performing  evaluation 
activities. 

Weakness 

Ability  to  perform  3 

Individuals  performing  evaluation 
activities  have  experience  or  receive 
training 

Project  team  members  performing 
evaluation  activities  have  experience  and 
have  received  training. 

Strength 

Ability  to  perform  4 

Members  of  the  project  team  and 
groups  supporting  the  software 
acquisition  receive  orientation  on  the 
objectives  of  the  evaluation  approach. 

Members  of  the  project  team  and  groups 
supporting  the  software  acquisition  did  not 
receive  orientation  on  the  objectives  of  the 
evaluation  approach. 

Weakness 

Activity  performed  1 

The  project  team  performs  its  activities 
in  accordance  with  its  documented 
evaluation  plans. 

There  is  no  evaluation  plan;  therefore,  the 
activities  are  not  performed  in  accordance 
with  it. 

Weakness 

Activity  performed  2 

The  project’s  evaluation  requirements 
are  developed  in  conjunction  with  the 
development  of  the  system  or  software 
technical  requirements. 

Project  evaluation  requirements  were 
developed  in  conjunction  with  software 
technical  requirements  and  are  designated 
in  the  statement  of  work. 

Strength 

Activity  performed  3 

The  project’s  evaluation  activities  are 
planned  to  minimize  duplication  and 
take  advantage  of  all  evaluation  results, 
where  appropriate. 

There  appears  to  be  no  duplication  of 
evaluation  activities. 

Strength 

Activity  performed  4 

The  project  team  appraises  the 
contractor  team’s  performance  over  the 
total  period  of  the  contract  for 
compliance  with  requirements. 

The  project  team  appraises  the  contractor 
team’s  performance  over  the  total  period  of 
the  contract  for  compliance  with 
requirements. 

Strength 

Activity  performed  5 

Planned  evaluations  are  performed  on 
the  evolving  software  products  and 
services  before  acceptance  for 
operational  use. 

Planned  evaluation  activities  are  performed 
on  the  evolving  software  products  and 
services  before  they  are  accepted  for 
operational  use. 

Strength 

Activity  performed  6 

Results  of  the  evaluations  are  analyzed 
and  compared  to  the  contract’s 
requirements  to  establish  an  objective 
basis  to  support  the  decision  to  accept 
the  product  or  services  or  to  take 
further  action. 

Results  of  the  evaluations  are  analyzed  and 
compared  to  the  contract’s  requirements  to 
establish  an  objective  basis  to  support  the 
decision  to  accept  the  software  products  or 
to  take  further  action. 

Strength 

Page  63 


GAO-Ol-962  HUD  Information  Systems 


Chapter  5:  HUD’s  Software  Evaluation 
Processes  Are  Not  Repeatable 


HUD’s  Central  Accounting  and  Program  System 

Common  feature 

Key  practice 

Finding 

Rating 

Measurement  and  analysis  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  evaluation 
activities  and  resultant  products. 

No  measurements  are  made  of  HUD’s 
evaluation  activities  or  resultant  products. 

Weakness 

Verifying  implementation  1 

Evaluation  activities  are  reviewed  by 
acquisition  organization  management 
on  a  periodic  basis. 

Acquisition  organization  management  does 
not  review  evaluation  activities  on  a  regular 
basis. 

Weakness 

Verifying  implementation  2 

Evaluation  activities  are  reviewed  by 
the  project  manager  on  both  a  periodic 
and  an  event-driven  basis. 

The  project  manager  stated  that  while  he 
discusses  the  status  of  the  project  with  team 
members,  meeting  results  and  action  items 
are  not  documented. 

Weakness 
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Table  22:  Software  Evaluation  Findings  for  the  Empowerment  Information  System 

Empowerment  Information  System 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  to  perform  1 

The  acquisition  organization  has  a 
written  policy  for  managing  the 
evaluation  of  acquired  software 
products  and  services. 

HUD  has  no  written  policy  for  managing  the 
evaluation  of  acquired  software  products  and 
services. 

Weakness 

Commitment  to  perform  2 

Responsibility  has  been  designated  for 
evaluation  activities. 

Responsibility  for  evaluation  activities  has 
been  assigned  to  the  project  lead. 

Strength 

Ability  to  perform  1 

A  group  is  responsible  for  planning, 
managing,  and  performing  evaluation 
activities. 

The  project  lead  and  team  members  are 
responsible  for  evaluation  activities. 

Strength 

Ability  to  perform  2 

Adequate  resources  are  provided  for 
evaluation  activities. 

There  is  no  mechanism  to  identify  needed 
resources  and  ensure  that  they  are  provided 
to  the  project.  The  project  manager  stated 
that  the  team  does  not  have  adequate 
resources  for  performing  evaluation  activities. 

Weakness 

Ability  to  perform  3 

Individuals  performing  evaluation 
activities  have  experience  or  receive 
training. 

Project  team  members  performing  evaluation 
activities  have  experience  and  have  received 
training. 

Strength 

Ability  to  perform  4 

Members  of  the  project  team  and 
groups  supporting  the  software 
acquisition  receive  orientation  on  the 
objectives  of  the  evaluation  approach. 

Members  of  the  project  team  or  others 
supporting  the  software  acquisition  did 
receive  orientation  on  the  objectives  of  the 
evaluation  approach. 

Strength 

Activity  performed  1 

The  project  team  performs  its  activities 
in  accordance  with  its  documented 
evaluation  plans. 

There  is  no  evaluation  plan;  therefore,  the 
activities  were  not  performed  in  accordance 
with  it. 

Weakness 

Activity  performed  2 

The  project’s  evaluation  requirements 
are  developed  in  conjunction  with  the 
development  of  the  system  or  software 
technical  requirements. 

The  project’s  evaluation  requirements  were 
not  developed  in  conjunction  with  the 
development  of  the  software  technical 
requirements. 

Weakness 

Activity  performed  3 

The  project’s  evaluation  activities  are 
planned  to  minimize  duplication  and 
take  advantage  of  all  evaluation 
results,  where  appropriate. 

There  is  no  evidence  of  evaluation  activities 
being  performed. 

Weakness 

Activity  performed  4 

The  project  team  appraises  the 
contractor  team’s  performance  over 
the  total  period  of  the  contract  for 
compliance  with  requirements. 

The  project  team  assessed  contractor  team 
performance  for  compliance  with 
requirements  only  at  the  end  of  the  effort. 

Weakness 

Activity  performed  5 

Planned  evaluations  are  performed  on 
the  evolving  software  products  and 
services  before  acceptance  for 
operational  use. 

No  planned  evaluations  were  performed  on 
the  software.  A  user  survey  was  conducted, 
and  the  software  was  rejected  on  the  basis  of 
the  survey. 

Weakness 

Activity  performed  6 

Results  of  the  evaluations  are 
analyzed  and  compared  to  the 
contract’s  requirements  to  establish  an 
objective  basis  to  support  the  decision 
to  accept  the  product  or  services  or  to 
take  further  action. 

No  planned  evaluations  were  conducted.  The 
software  was  rejected  based  on  the  results  of 
the  user  survey. 

Weakness 
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Empowerment  Information  System 

Common  feature 

Key  practice 

Finding 

Rating 

Measurement  and  analysis  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  evaluation 
activities  and  resultant  products. 

No  measurements  are  made  of  HUD’s 
evaluation  activities  or  resultant  products. 

The  cost  of  the  user  survey  was  reported. 

Observation 

Verifying  implementation  1 

Evaluation  activities  are  reviewed  by 
acquisition  organization  management 
on  a  periodic  basis. 

Acquisition  organization  management  does 
not  review  evaluation  activities  on  a  regular 
basis. 

Weakness 

Verifying  implementation  2 

Evaluation  activities  are  reviewed  by 
the  project  manager  on  both  a  periodic 
and  an  event-driven  basis. 

The  project  manager  does  not  formally 
review  evaluation  activities,  and  the  project 
manager’s  reviews  are  not  documented. 

Weakness 
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HUD’s  software  acquisition  processes  are  immature  and  are  not  repeatable 
on  a  project-by-project  basis  because  of  the  many  weaknesses  it  has  in  the 
key  practices.  Strong  performance  of  the  key  practices  is  essential  for 
achieving  effective,  repeatable,  and  lasting  implementation  and 
institutionalization  of  the  four  key  process  areas  we  reviewed.  Currently, 
HUD’s  success  or  failure  in  acquiring  software  depends  largely  on  specific 
individuals,  rather  than  on  well-defined  and  disciplined  software 
acquisition  management  practices.  As  a  result,  HUD  is  exposed  to  higher 
risks  that  software-intensive  acquisition  projects  will  not  consistently  meet 
mission  requirements,  perform  as  intended,  or  be  delivered  on  schedule 
and  within  budget. 

HUD  acknowledged  that  it  has  software  acquisition  weaknesses  and  is 
committed  to  improving  its  software  and  system  acquisition  processes.  We 
agree  that  HUD  should  begin  a  software  process  improvement  effort. 
However,  for  HUD  to  successfully  complete  such  an  effort,  its  software 
process  improvement  plan  must  address  several  important  issues.  To  be 
comprehensive,  this  plan  should  include  the  results  of  our  review  and 
include  measurable  goals  and  time  frames,  estimates  of  resource 
requirements,  and  time  frames  to  reach  the  repeatable  level.  It  should  also 
contain  steps  to  address  the  systemic  weaknesses  we  found.  Finally,  the 
plan  should  include  steps  to  address  the  areas  we  did  not  review  and  to 
ensure  that  all  ongoing  as  well  as  new  software  acquisition  projects  adopt 
processes  and  meet  the  requirements  for  the  repeatable  level  of  maturity. 

To  improve  HUD’s  software  acquisition  capabilities,  GAO  recommends 
that  the  Secretary  of  Housing  and  Urban  Development  direct  the  HUD 
Chief  Information  Officer  to  develop  and  implement  a  comprehensive  plan 
for  software  acquisition  process  improvement  that  is  based  on  the 
software  capability  results  in  this  report  and  specifies  measurable  goals 
and  time  frames,  sets  priorities  for  initiatives,  estimates  resource 
requirements  (for  trained  staff  and  funding),  and  defines  a  process 
improvement  management  structure. 

Also,  to  address  the  systemic  weaknesses  mentioned  above,  the  plan 
should  contain  steps  to 

•  develop  a  comprehensive  policy  for  the  acquisition  of  software, 

•  require  plans  for  specific  acquisition  activities, 

•  measure  and  track  the  status  of  activities  performed  by  the  software 
acquisition  teams,  and 

•  report  regularly  to  management  on  progress  and  problems. 
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Agency  Comments 
and  Our  Evaluation 


In  addition,  to  ensure  that  all  aspects  of  software  acquisition  are 
addressed,  we  recommend  that  the  Secretary  direct  the  CIO  to 

assess  HUD’s  maturity  in  the  three  key  process  areas  that  could  not  be 
evaluated  by  GAO  and  include  any  needed  improvement  actions  in  the 
comprehensive  plan  for  software  acquisition  process  improvement; 
ensure  that  all  new  software  acquisition  projects  in  HUD  adopt  processes 
that  satisfy  at  least  SA-CMM  level  2  requirements;  and 
ensure  that  process  improvement  activities  are  initiated  for  all  ongoing 
software  acquisition  projects. 


In  providing  written  comments  on  a  draft  of  this  report,  the  Chief 
Information  Officer  (CIO)  stated  that  HUD  concurred  with  our 
recommendations  to  strengthen  its  software  acquisition  processes.  The 
CIO  acknowledged  that  its  software  acquisition  processes  have 
weaknesses  and  stated  that  it  has  an  improvement  project  under  way.  This 
project  includes  plans  to  seek  contractor  assistance  to  address  the 
weaknesses  we  identified  and  help  improve  HUD’s  software  acquisition 
processes.  Finally,  the  CIO  stated  that  the  department  generally  agreed 
with  our  assessment  of  its  software  acquisition  practices.  HUD  also 
provided  additional  documentation  on  its  practices  for  the  Resident 
Assessment  Subsystem.  We  reviewed  this  information  and  made  revisions 
where  appropriate  and  supported  by  the  additional  documentation.  The 
department’s  comments  are  reprinted  as  appendix  I. 
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U.S.  DEPARTMENT  OF  HOUSING  AND  URBAN  DEVELOPMENT 
WASHINGTON,  D.C.  20410-3000 

August  17,  2001 


Mr.  Joel  Willemssen 
Managing  Director 
Information  Technology  Issues 
General  Accounting  Office 
Washington,  DC  20548 


Dear  Mr.  Willemssen: 

This  is  in  response  to  your  letter  to  Secretary  Martinez  dated  August  3, 2001, 
transmitting  the  GAO  Draft  Report:  HUD  Information  Systems:  Immature  Software 
Acquisition  Capability  Increases  Project  Risks  (CIO-1526/5 1 1 866).  Your  letter 
requested  comments  on  the  draft  report  within  14  days. 

HUD  previously  recognized  weaknesses  in  the  area  of  software  acquisition  and  has 
an  active  Software  Acquisition  Capability  Maturity  Model  (SA-CMM)  Improvement 
Project  already  underway.  HUD’s  plan  is  to  award  a  contract  to  a  CMM  level  3  or  higher 
company  that  will  assist  HUD  in  addressing  the  weaknesses  in  the  GAO  report,  along 
with  the  three  key  process  areas  that  could  not  be  evaluated  by  GAO,  ultimately 
achieving  compliance  with  SA-CMM  level  2. 

HUD  agrees  with  the  draft  recommendations.  While  HUD  agrees  that  an 
aggressive  improvement  plan  is  needed,  HUD  believes  that  a  systematic,  planned 
approach  to  the  following  recommendations  is  critical  for  success: 

•  Ensure  that  all  new  software  acquisition  projects  adopt  processes  that  satisfy 
at  least  SA-CMM  level  2  requirements. 

•  Ensure  that  process  improvement  activities  are  initiated  for  all  ongoing 
software  acquisition  projects. 

HUD  will  include  implementation  schedules  for  these  two  recommendations  in  the 
comprehensive  improvement  plan. 

All  appropriate  HUD  program  offices  have  reviewed  the  draft  report.  All  generally 
agree  with  the  findings  except  for  the  Real  Estate  Assessment  Center  (REAC)  which  is 
currently  working  with  GAO  to  resolve  proposed  weaknesses  with  the  Resident 
Assessment  Subsystem  (RASS)  in  the  areas  of  Requirements  Development  and 
Management,  Project  Management,  Contract  Tracking  and  Oversight,  and  Evaluation. 


CHIEF  INFORMATION  OFFICER 
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If  you  or  your  staff  have  any  questions,  please  contact  Pamela  Woodside,  Director, 
Office  of  Systems  Integration  and  Efficiency,  on  (202)  708-2050. 

Sincerely, 


Gloria  R.  Parker 
Chief  Information  Officer 

cc: 

James  M.  Martin,  Deputy  Assistant  Chief  Financial  Officer  for  Financial  Management 
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